clewdex 0.1.0__tar.gz

clewdex-0.1.0/.claude/settings.local.json

@@ -0,0 +1,65 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(python -m pytest:*)",
+      "Bash(python3 -m pytest:*)",
+      "Bash(pytest:*)",
+      "Bash(uv run pytest:*)",
+      "Bash(python3 -m ruff check:*)",
+      "Bash(python3 -m ruff format:*)",
+      "Bash(ruff check:*)",
+      "Bash(ruff format:*)",
+      "Bash(mypy:*)",
+      "Bash(git branch:*)",
+      "Bash(python3:*)",
+      "Bash(git filter-branch:*)",
+      "Bash(FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch:*)",
+      "Bash(grep:*)",
+      "Bash(find:*)",
+      "Bash(git fsck:*)",
+      "WebSearch",
+      "Bash(gh search:*)",
+      "WebFetch(domain:pypi.org)",
+      "WebFetch(domain:www.npmjs.com)",
+      "WebFetch(domain:github.com)",
+      "WebFetch(domain:formulae.brew.sh)",
+      "Bash(pip show:*)",
+      "Bash(pip uninstall:*)",
+      "Bash(git mv:*)",
+      "Bash(pip install:*)",
+      "Bash(clew --help:*)",
+      "mcp__plugin_episodic-memory_episodic-memory__search",
+      "mcp__plugin_episodic-memory_episodic-memory__read",
+      "mcp__plugin_context7_context7__resolve-library-id",
+      "mcp__plugin_context7_context7__query-docs",
+      "WebFetch(domain:weaviate.io)",
+      "WebFetch(domain:sourcegraph.com)",
+      "WebFetch(domain:www.greptile.com)",
+      "WebFetch(domain:towardsdatascience.com)",
+      "WebFetch(domain:read.engineerscodex.com)",
+      "WebFetch(domain:blog.voyageai.com)",
+      "WebFetch(domain:www.infoq.com)",
+      "WebFetch(domain:qdrant.tech)",
+      "WebFetch(domain:arxiv.org)",
+      "WebFetch(domain:news.ycombinator.com)",
+      "WebFetch(domain:medium.com)",
+      "WebFetch(domain:registry.npmjs.org)",
+      "WebFetch(domain:crates.io)",
+      "Bash(curl:*)",
+      "Bash(python -m build:*)",
+      "Bash(node:*)",
+      "Bash(clew index:*)",
+      "Bash(QDRANT_URL=http://localhost:6333 clew index:*)",
+      "Bash(QDRANT_URL=http://localhost:6333 clew search:*)",
+      "Bash(QDRANT_URL=http://localhost:6333 clew trace:*)",
+      "Bash(clew status:*)",
+      "Bash(QDRANT_URL=http://localhost:6333 clew status:*)",
+      "Bash(clew trace:*)",
+      "Bash(clew search:*)",
+      "Bash(sqlite3:*)",
+      "Bash(python -m clew.cli:*)",
+      "Bash(clew reembed:*)",
+      "Bash(QDRANT_URL=http://localhost:6333 clew reembed:*)"
+    ]
+  }
+}

clewdex-0.1.0/.claude/skills/clew-enrich/SKILL.md

@@ -0,0 +1,166 @@
+# /clew-enrich
+
+Enrich indexed code chunks with LLM-generated descriptions and keywords to improve semantic search quality.
+
+## When to Use
+
+Run `/clew-enrich` after `clew index` has completed. Enrichment adds natural language descriptions and search keywords to each code chunk, enabling better semantic search results for natural language queries like "how does order processing work".
+
+## Prerequisites
+
+- The project must already be indexed (`clew index [PROJECT_ROOT]`)
+- The `.clew/` directory must exist in the project root (created by `clew index`)
+
+## What It Does
+
+1. Reads all chunk IDs from the SQLite cache (`cache.db`)
+2. Identifies chunks that have NOT yet been enriched
+3. For each unenriched chunk, reads the source code from disk
+4. Generates a description (2-3 sentences) and keywords (8-15 terms) per chunk
+5. Writes enrichment data back to the SQLite cache
+6. Triggers `clew reembed` to re-embed all enriched chunks with the new content
+
+## Step-by-Step Instructions
+
+### Step 1: Locate the Cache Directory
+
+The cache directory is at `{project_root}/.clew/`. The project root is typically the git root. If `CLEW_CACHE_DIR` is set, use that path instead.
+
+```bash
+# Find the git root
+git rev-parse --show-toplevel
+```
+
+The cache database file is `{cache_dir}/cache.db`.
+
+### Step 2: Read Unenriched Chunks
+
+Run this Python script to get the list of unenriched chunk IDs and their file paths:
+
+```python
+import sqlite3
+import json
+
+cache_dir = "{PROJECT_ROOT}/.clew"  # Replace with actual path
+conn = sqlite3.connect(f"{cache_dir}/cache.db")
+conn.row_factory = sqlite3.Row
+
+# Get all chunk_ids from chunk_cache
+rows = conn.execute("SELECT file_path, chunk_ids FROM chunk_cache").fetchall()
+all_chunks = []
+for row in rows:
+    file_path = row["file_path"]
+    chunk_ids = json.loads(row["chunk_ids"])
+    for cid in chunk_ids:
+        all_chunks.append((cid, file_path))
+
+# Get already-enriched chunk_ids
+enriched = set(
+    r[0] for r in conn.execute("SELECT chunk_id FROM enrichment_cache").fetchall()
+)
+conn.close()
+
+# Filter to unenriched
+unenriched = [(cid, fp) for cid, fp in all_chunks if cid not in enriched]
+print(f"Total chunks: {len(all_chunks)}")
+print(f"Already enriched: {len(enriched)}")
+print(f"Unenriched: {len(unenriched)}")
+```
+
+### Step 3: Enrich Each Chunk
+
+For each unenriched chunk, you need to:
+
+1. **Parse the chunk_id** to extract entity info. The format is: `file_path::entity_type::qualified_name` (for named entities) or `file_path::toplevel::sha256hash` (for anonymous/toplevel chunks).
+
+2. **Read the source file** from disk using the `file_path` from the chunk_id (the part before the first `::`).
+
+3. **Generate description and keywords**. For each chunk, produce output in this exact format:
+   ```
+   Description: 2-3 sentences explaining what the code does, why it exists, and what domain concept it represents.
+   Keywords: 8-15 space-separated terms a developer might search for when looking for this code.
+   ```
+
+   When generating descriptions and keywords, consider:
+   - What does this code do at a high level?
+   - What domain concepts does it relate to?
+   - What would someone search for to find this code?
+   - Include both technical terms and domain-specific vocabulary
+
+4. **Write enrichment to cache** using this Python script (batch mode):
+
+```python
+import sqlite3
+import time
+
+cache_dir = "{PROJECT_ROOT}/.clew"  # Replace with actual path
+conn = sqlite3.connect(f"{cache_dir}/cache.db")
+
+enrichments = [
+    # ("chunk_id", "description text", "keyword1 keyword2 keyword3 ..."),
+]
+
+for chunk_id, description, keywords in enrichments:
+    conn.execute(
+        "INSERT OR REPLACE INTO enrichment_cache "
+        "(chunk_id, description, keywords, enriched_at) "
+        "VALUES (?, ?, ?, ?)",
+        (chunk_id, description, keywords, time.time()),
+    )
+
+conn.commit()
+conn.close()
+print(f"Wrote {len(enrichments)} enrichments to cache")
+```
+
+### Step 4: Process in Batches
+
+Process chunks in batches of 20-50 to manage context window size:
+
+1. Read a batch of source files using the Read tool
+2. Generate descriptions and keywords for all chunks in the batch
+3. Write the batch to SQLite
+4. Repeat until all chunks are processed
+
+Skip chunks where:
+- The file no longer exists on disk
+- The chunk_id contains `::toplevel::` (anonymous chunks have less value to enrich)
+- The chunk_id contains `::file_summary::` (already synthetic)
+
+### Step 5: Re-embed
+
+After all enrichments are written to the cache, run:
+
+```bash
+clew reembed {PROJECT_ROOT}
+```
+
+This reads the enrichment data from cache and re-embeds all enriched chunks with the full content (description + keywords + code) into Qdrant's named vectors.
+
+## Enrichment Cache Schema
+
+The enrichment data is stored in `{cache_dir}/cache.db` in the `enrichment_cache` table:
+
+```sql
+CREATE TABLE IF NOT EXISTS enrichment_cache (
+    chunk_id TEXT PRIMARY KEY,
+    description TEXT,
+    keywords TEXT,
+    enriched_at REAL
+);
+```
+
+## Example Output
+
+For a chunk with ID `backend/ecomm/utils.py::function::EcommUtils._process_shopify_order_impl`:
+
+```
+Description: Processes incoming Shopify orders by validating order data, creating internal Order records, and triggering fulfillment workflows. This is the core order ingestion handler called by the Shopify webhook receiver.
+Keywords: shopify order processing webhook fulfillment ecommerce cart purchase payment order_ingestion
+```
+
+## Notes
+
+- Enrichment is idempotent: running `/clew-enrich` again skips already-enriched chunks
+- To re-enrich everything, delete the `enrichment_cache` table contents first: `DELETE FROM enrichment_cache` in `cache.db`
+- The `clew reembed` step is required after writing enrichments; without it, the search index won't reflect the new descriptions

clewdex-0.1.0/.env.example

@@ -0,0 +1,8 @@
+# Required
+VOYAGE_API_KEY=pa-xxxxxxxxxxxxxxxxxxxx
+
+# Optional (defaults shown)
+QDRANT_URL=http://localhost:6333
+QDRANT_API_KEY=
+CLEW_CACHE_DIR=.clew
+CLEW_LOG_LEVEL=INFO

clewdex-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,37 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Lint
+        run: ruff check .
+
+      - name: Format check
+        run: ruff format --check .
+
+      - name: Type check
+        run: mypy clew/
+
+      - name: Test
+        run: pytest --cov=clew -v --ignore=tests/integration

clewdex-0.1.0/.github/workflows/publish-npm.yml

@@ -0,0 +1,25 @@
+name: Publish to npm
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: npm
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+          registry-url: "https://registry.npmjs.org"
+
+      - name: Publish to npm
+        run: npm publish --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

clewdex-0.1.0/.github/workflows/publish-pypi.yml

@@ -0,0 +1,47 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  id-token: write
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tools
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

clewdex-0.1.0/.gitignore

@@ -0,0 +1,31 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.egg-info/
+dist/
+build/
+.venv/
+venv/
+
+# IDE
+.vscode/
+.idea/
+
+# Environment
+.env
+
+# Cache
+.code-search/
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+
+# Qdrant local data
+.qdrant/
+
+# Worktrees
+.worktrees/
+
+# Coverage
+.coverage

clewdex-0.1.0/CLAUDE.md

@@ -0,0 +1,156 @@
+# clew
+
+Semantic code search tool with hybrid retrieval and MCP integration for Claude Code.
+
+## Project State
+
+**Phase 1 (Core Infrastructure) is complete.** 20 source modules, 10 test files, 80 tests passing at 86% coverage.
+
+**Phase 2 (Search Pipeline) is complete.** 27 source modules, 16 test files, 240 tests passing at 91% coverage.
+
+**Phase 3 (MCP Integration & CLI) is complete.** 31 source modules, 27 test files, 349 tests passing at 92% coverage.
+
+**V1.1 (NL Descriptions) is complete.** 32 source modules, 30 test files, 394 tests passing. LLM-generated descriptions for undocumented code chunks, prepended before embedding to improve semantic search quality.
+
+**V1.2 (Structural Layer) is complete.** 39 source modules, 36 test files, 472 tests passing. Code relationship extraction (imports, inherits, calls, decorates, renders, tests, calls_api) with BFS graph traversal via `trace` MCP tool and CLI command.
+
+**V1.3 (Compact Responses & Cache Fix) is complete.** 39 source modules, 36 test files, 491 tests passing. Compact MCP responses by default (~20x token reduction), opt-in full content via `detail="full"`. CACHE_DIR now resolves from git root so MCP server and indexer share the same state.db.
+
+## Module Inventory
+
+```
+clew/
+├── chunker/            # AST parsing (tree-sitter), language strategies, token-aware fallback splitting
+│   ├── parser.py       # ASTParser — tree-sitter wrapper, language detection by extension
+│   ├── strategies.py   # PythonChunker — extracts functions/classes as CodeEntity dataclasses
+│   ├── fallback.py     # Token-recursive + line-based splitting; Chunk dataclass with metadata dict
+│   └── tokenizer.py    # tiktoken cl100k_base token counting
+├── clients/            # External service abstractions
+│   ├── base.py         # EmbeddingProvider ABC (embed, embed_query, dimensions, model_name)
+│   ├── description.py  # DescriptionProvider ABC + AnthropicDescriptionProvider — NL descriptions for code
+│   ├── qdrant.py       # QdrantManager — collection CRUD, hybrid query with RRF fusion, delete by file_path
+│   └── voyage.py       # VoyageEmbeddingProvider — httpx async client for Voyage AI
+├── indexer/            # File discovery, caching, change detection, indexing pipeline, relationship extraction
+│   ├── cache.py        # CacheDB — SQLite via contextmanager, embedding + chunk caches, state tracking, relationship store
+│   ├── change_detector.py # ChangeDetector — unified interface: git-first, file-hash fallback
+│   ├── extractors/     # Pluggable relationship extractors (V1.2)
+│   │   ├── base.py     # RelationshipExtractor ABC
+│   │   ├── python.py   # Python: imports, inherits, decorates, calls
+│   │   ├── typescript.py # TypeScript/JS: imports, inherits, renders (JSX), calls, calls_api (fetch/axios)
+│   │   ├── tests.py    # Test file detection: maps test files to tested modules
+│   │   ├── django_urls.py # Django URL pattern extraction from urls.py
+│   │   └── api_boundary.py # Cross-language API boundary matching (frontend→backend)
+│   ├── file_hash.py    # FileHashTracker — SHA256-based change detection (added/modified/unchanged)
+│   ├── git_tracker.py  # GitChangeTracker — git diff --name-status change detection (A/M/D/R parsing)
+│   ├── ignore.py       # IgnorePatternLoader — .gitignore + .clewignore + defaults
+│   ├── metadata.py     # detect_app_name, classify_layer, extract_signature, build_chunk_id
+│   ├── pipeline.py     # IndexingPipeline — file -> chunk -> metadata -> embed -> upsert + relationship extraction
+│   └── relationships.py # Relationship dataclass — entity-relationship-entity with confidence
+├── search/             # Search pipeline: enhance -> classify -> hybrid search -> rerank
+│   ├── engine.py       # SearchEngine — top-level orchestrator, full pipeline coordination
+│   ├── enhance.py      # QueryEnhancer — terminology expansion from YAML (abbreviations + synonyms)
+│   ├── hybrid.py       # HybridSearchEngine — dense + BM25 multi-prefetch with structural boosting
+│   ├── intent.py       # classify_intent — keyword heuristic intent routing (DEBUG > LOCATION > DOCS > CODE)
+│   ├── models.py       # QueryIntent, SearchResult, SearchRequest, SearchResponse dataclasses
+│   ├── filters.py      # build_qdrant_filter() — converts SearchRequest.filters to Qdrant Filter objects
+│   ├── rerank.py       # RerankProvider — Voyage rerank-2.5 integration with configurable skip conditions
+│   └── tokenize.py     # BM25 tokenization — camelCase/snake_case splitting, raw term count sparse vectors
+├── cli.py              # Typer app — index, search, status, trace, serve commands (fully wired)
+├── config.py           # Environment class — env var loading with defaults
+├── discovery.py        # discover_files() — centralized file discovery using IgnorePatternLoader + SafetyChecker
+├── exceptions.py       # Exception hierarchy with user-facing fix suggestions
+├── factory.py          # Component factory — centralized wiring, create_components() returns Components dataclass
+├── mcp_server.py       # FastMCP server — 5 tools: search, get_context, explain, index_status, trace
+├── models.py           # Pydantic v2 models — ProjectConfig, SearchConfig, CollectionConfig, SafetyConfig, etc.
+└── safety.py           # SafetyChecker — file size, chunk count, collection limits
+```
+
+## Established Patterns
+
+- **Data models:** Pydantic v2 `BaseModel` for config/validation, `@dataclass` for internal data (CodeEntity, FileChange, SearchResult)
+- **Provider abstraction:** ABC base classes (e.g., `EmbeddingProvider`) with concrete implementations
+- **SQLite access:** `contextmanager` pattern in `CacheDB._connect()` — no ORM
+- **Config:** `Environment` class reads env vars with sensible defaults; `ProjectConfig` loaded from YAML
+- **Exceptions:** Hierarchy rooted in `ClewError`, each with `fix_hint` for user-facing messages
+- **Async:** Used for external API calls (Voyage AI, Qdrant hybrid search, embedding); sync for file I/O and SQLite
+- **Deterministic IDs:** Qdrant point IDs are UUID5 derived from structured chunk IDs (format: `file_path::entity_type::qualified_name`)
+
+## Commands
+
+```bash
+# Dev commands
+pytest --cov=clew -v    # Run tests with coverage
+pytest -m integration          # Run integration tests only
+ruff check .                   # Lint
+ruff format --check .          # Check formatting
+mypy clew/              # Type check
+
+# CLI commands
+clew index [PROJECT_ROOT] --full    # Full reindex
+clew index [PROJECT_ROOT]           # Incremental (change detection)
+clew index [PROJECT_ROOT] --nl-descriptions  # Generate NL descriptions (requires ANTHROPIC_API_KEY)
+clew search "query" --raw           # Search with JSON output
+clew trace "entity::name"           # Trace code relationships (BFS graph traversal)
+clew trace "entity" --direction outbound --depth 3  # Directed trace with depth limit
+clew status                          # Show Qdrant health + index stats
+clew serve                           # Start MCP server (stdio transport)
+```
+
+## Key Files
+
+- `docs/DESIGN.md` — Architecture, chunking strategy, search pipeline, MCP tools, metadata schema
+- `docs/IMPLEMENTATION.md` — Concrete specs: dependencies, SQLite schemas, Pydantic models, tree-sitter setup, phase tasks
+- `docs/adr/` — Architecture Decision Records (Qdrant over Milvus, build vs adopt)
+- `docs/plans/2026-02-06-phase1-core-infrastructure.md` — Phase 1 plan (complete)
+- `docs/plans/2026-02-06-phase2-search-pipeline.md` — Phase 2 plan (complete)
+- `docs/plans/2026-02-09-v1.1-nl-descriptions.md` — V1.1 NL Descriptions plan (complete)
+- `docs/plans/2026-02-09-v1.2-structural-layer.md` — V1.2 Structural Layer plan (complete)
+- `docs/plans/2026-02-10-compact-responses-and-cache-fix.md` — V1.3 Compact Responses & Cache Fix plan (complete)
+- `docs/plans/2026-02-06-three-layer-knowledge-design.md` — Future roadmap (V1.4+)
+
+## Tech Stack
+
+- Python >=3.10, Qdrant (Docker), Voyage AI voyage-code-3, tree-sitter, typer + rich CLI, Pydantic v2, SQLite for caching
+- Testing: pytest + pytest-asyncio, respx for HTTP mocking
+- Linting: ruff, mypy (strict)
+
+## MCP Server Configuration
+
+Add to Claude Code's `.mcp.json`:
+```json
+{
+  "mcpServers": {
+    "clew": {
+      "command": "clew",
+      "args": ["serve"],
+      "env": {
+        "VOYAGE_API_KEY": "your-key-here",
+        "QDRANT_URL": "http://localhost:6333",
+        "ANTHROPIC_API_KEY": "your-key-here (optional, for NL descriptions)",
+        "CLEW_CACHE_DIR": "/absolute/path/to/project/.clew (optional, auto-detected from git root)"
+      }
+    }
+  }
+}
+```
+
+## MCP Tool Response Modes
+
+MCP tools default to **compact** responses to minimize context window usage:
+
+- **`search`** — Returns `snippet` (signature + docstring preview) instead of full source. Default `limit=5`. Pass `detail="full"` for complete content.
+- **`explain`** — Same compact/full behavior. Default `limit=5`.
+- **`get_context`** — Returns file content only. Pass `include_related=True` to also get related code chunks (compact format).
+- **`trace`** and **`index_status`** — Already compact, no changes needed.
+
+The agent can always use the `Read` tool to fetch specific lines from results that look promising.
+
+## Conventions
+
+- Use `ruff` for formatting and linting
+- Use `mypy --strict` for type checking
+- Async where interacting with Voyage API or Qdrant
+- All config through Pydantic models validated from YAML
+- Error messages should tell the user how to fix the problem (e.g., "Qdrant not running. Start with: docker compose up -d qdrant")
+- Component wiring through `factory.py` — no global state, one factory call per invocation
+- MCP tools return structured dicts with `error` + `fix` keys on failure