archex 0.1.0.dev0__tar.gz → 0.4.0__tar.gz

{archex-0.1.0.dev0 → archex-0.4.0}/.github/workflows/ci.yml

@@ -23,9 +23,11 @@ jobs:
 
       - name: Install uv
         uses: astral-sh/setup-uv@v5
+        with:
+          python-version: ${{ matrix.python-version }}
 
       - name: Install dependencies
-        run: uv sync --dev
+        run: uv sync --all-extras
 
       - name: Lint
         run: uv run ruff check .

archex-0.4.0/CHANGELOG.md

@@ -0,0 +1,148 @@
+# Changelog
+
+## 0.4.0 (2026-03-01)
+
+### Refactoring
+
+- **Shared tree-sitter helpers:** Extract duplicate `_text`/`_type`/`_children`/`_field`/`_start_line`/`_end_line` accessors from all four language adapters into a single `ts_node` module
+- **Dead code removal:** Remove unused `get_adapter()`, `_extract_interfaces()`, redundant `add_node` calls in `DependencyGraph.from_edges()`, unused `index_config` param from `api.analyze()`
+- **Dependency deduplication:** Use sets instead of lists in `_build_module_from_community` for O(1) membership checks
+- **Chunker optimization:** Remove unnecessary `sorted()` call on already-ordered covered ranges
+- **Vector load:** `copy=False` on `numpy.astype` for zero-copy when array is already float32
+
+### Error handling & logging
+
+- **`infer_decisions()`:** Log LLM enrichment failures with `logger.warning()` instead of silently catching
+- **`BM25Index.search()`:** Log FTS5 query failures instead of silently returning empty results
+- **`SentenceTransformerEmbedder.dimension`:** Replace bare `assert` with explicit `ArchexIndexError`
+
+### Configuration & standards
+
+- **`DEFAULT_CACHE_DIR` constant:** Centralized in `config.py`, used by all CLI cache commands
+- **Config validation:** `model_fields` over `hasattr` for Pydantic v2 correctness
+- **`validate_dimensions()`:** Extracted from `compare_repos()` for reuse by MCP integration
+- **Install instructions:** All `pip install` references replaced with `uv add`
+
+### Testing
+
+- 3 new test files: `test_config.py`, `test_adapter_registry.py`, `test_renderers.py`
+- 7 extended test files covering parse, index, analyze, serve, and acquire layers
+- 538 → 641 tests (+103), 84% → 90% coverage
+
+## 0.3.0 (2026-03-01)
+
+### Phase 6a — Harden
+
+- **Git URL validation:** `_validate_url()` restricts to `http://`, `https://`, local paths only
+- **Branch name validation:** Regex guard rejects injection characters and `-` prefix
+- **FTS5 query escaping:** Strip non-alphanumeric characters from BM25 query tokens
+- **Cache key validation:** Enforce `^[0-9a-f]{64}$` pattern in `db_path()` and `meta_path()`
+- **Vector safety:** `allow_pickle=False` and `dtype='U512'` for `.npz` persistence, length validation on load
+- **File size guard:** `max_file_size` config in `discover_files()` and `parse_file()`
+- **Store safety:** `IndexStore.__init__` wrapped in try/except for connection cleanup on failure
+- **Parse logging:** `symbols.py` and `imports.py` log warnings on parse failures
+- **MCP validation:** Dimension list validated against `SUPPORTED_DIMENSIONS` before `compare()`
+- **MCP event loop:** `asyncio.get_event_loop()` → `asyncio.get_running_loop()`
+- **CLI error handling:** API calls wrapped in `try/except ArchexError` → `click.ClickException`
+- **Embeddings timeout:** `timeout=30` added to API `urlopen()`
+- **Compare CLI:** `assert isinstance(...)` replaced with explicit type check
+
+### Phase 6b — Performance
+
+- **Cache-first query:** `query()` checks cache BEFORE parsing — cache hit skips entire parse pipeline
+- **Graph round-trip:** `DependencyGraph.from_edges()` classmethod reconstructs graph from stored edges
+- **Batch fetch:** `IndexStore.get_chunks_by_ids()` with `WHERE id IN (...)`, used in `BM25Index.search()`
+- **Parallel config:** `Config.parallel` flag passed to `extract_symbols()` and `parse_imports()`
+- **Parallel compare:** `ThreadPoolExecutor(max_workers=2)` runs both `analyze()` calls concurrently
+- **O(N) top-k:** `np.argpartition` replaces `np.argsort` in VectorIndex search
+- **Vector cache:** `CacheManager.vector_path()` persists vector indices across queries
+- **Centrality cache:** Lazy `_centrality_cache` on `DependencyGraph`, invalidated on mutation
+- **Chunker optimization:** Source split once in `chunk_file()`, pre-split lines passed downstream
+- **Git-aware cache:** Cache key includes git HEAD commit hash for local repos
+
+### Phase 6c — Wire & Polish
+
+- **Hybrid retrieval wired:** VectorIndex built and searched in cache-miss query path, results passed through RRF to `assemble_context()`
+- **`resolve_source()` utility:** Extracted from 4 inline copies, fixes `query_cmd` bug (`startswith("http")` → `startswith("http://")`)
+- **Compare CLI routing:** Routes through `api.compare()` instead of manual `analyze()` x2
+- **MCP dimension fix:** `testing_strategy` → `testing`, `dependency_management` → `state_management`, `configuration_management` → `configuration`
+- **Dead field removal:** `CodeChunk.module` removed from models, store schema, and chunker
+- **RepoSource validator:** `model_validator(mode="after")` requires `url` or `local_path`
+- **`load_config()`:** Reads `~/.archex/config.toml` via `tomllib` + `ARCHEX_*` env vars
+- **Provider model IDs:** Centralized in `DEFAULT_MODELS` dict in `config.py`
+- **Pipeline logging:** `logging.getLogger(__name__)` with timing at all stage boundaries
+- **Test improvements:** Cache CLI tests, `__version__` import in test_cli
+
+### Phase 6d — Extensibility
+
+- **`ScoringWeights` model:** Parameterized context scoring (relevance=0.6, structural=0.3, type_coverage=0.1) with sum-to-1 validator, accepted in `assemble_context()` and `query()`
+- **`PatternRegistry`:** `register()` decorator, `load_entry_points()` for `archex.pattern_detectors` group, optional `registry` param in `detect_patterns()`
+- **`AdapterRegistry`:** `register()`, `build_all()`, `load_entry_points()` for `archex.language_adapters` group, public `adapter_classes` property
+- **`Chunker` Protocol:** `runtime_checkable`, accepted as optional `chunker` param in `query()`
+- **Entry points:** `archex.language_adapters` and `archex.pattern_detectors` groups declared in `pyproject.toml`
+- **Integration tests:** 12 end-to-end tests covering analyze, query (BM25, caching, custom weights, hybrid fallback), compare (default + specific dimensions), full analyze→query pipeline
+- 538 tests, 84% coverage
+
+## 0.2.0 (2026-02-28)
+
+### Phase 5 — Ecosystem
+
+- **MCP server:** 3 tools (analyze_repo, query_repo, compare_repos) with async stdio transport, `archex mcp` CLI command
+- **LangChain integration:** `ArchexRetriever(BaseRetriever)` mapping RankedChunks to Documents
+- **LlamaIndex integration:** `ArchexRetriever(BaseRetriever)` mapping RankedChunks to NodeWithScore
+- **Parallel parsing:** `extract_symbols()` and `parse_imports()` accept `parallel=True` for ProcessPoolExecutor concurrency
+- **ONNX model caching:** `NomicCodeEmbedder` supports `cache_dir` for persistent model storage at `~/.archex/models/`
+- **New optional deps:** `archex[mcp]`, `archex[langchain]`, `archex[llamaindex]`
+- 422 tests, 81% coverage
+
+## 0.1.0 (2026-02-28)
+
+### Phase 0 — Scaffold
+
+- Project structure with hatchling build system, CLI stub, CI config
+- Test fixtures: `python_simple`, `python_patterns`, `typescript_simple`, `monorepo_simple`
+- Tooling: ruff, pyright (strict), pytest + pytest-cov, pre-commit
+
+### Phase 1 — Foundation
+
+- **Acquire:** `clone_repo()`, `open_local()`, `discover_files()` with git ls-files + rglob fallback
+- **Parse:** `TreeSitterEngine` with cached Language/Parser, `PythonAdapter` (full AST walk)
+- **Index:** `DependencyGraph` wrapping dual NetworkX DiGraphs (file-level, symbol-level), SQLite round-trip, PageRank centrality
+- **Serve:** Basic `ArchProfile` assembly with stats, dependency summary, interface surface
+- **API:** `analyze()` pipeline — discover, parse, resolve, graph, profile
+- **CLI:** `archex analyze <source> --format json|markdown`
+- **Models:** Complete Pydantic v2 model hierarchy, exception classes, StrEnum types
+- 107 tests, 88% coverage
+
+### Phase 2 — Retrieval
+
+- **Chunker:** `ASTChunker` with symbol-based boundaries, import context, small-chunk merging, tiktoken counting
+- **BM25:** SQLite FTS5 index with OR-joined query tokens
+- **Store:** `IndexStore` with WAL mode, chunks/edges/metadata tables
+- **Cache:** `CacheManager` with TTL, WAL checkpoint before copy, FTS rebuild on load
+- **Context assembly:** BM25 search, graph neighborhood expansion, composite scoring (0.6 relevance + 0.3 structural + 0.1 type), greedy bin-packing
+- **Renderers:** XML (CDATA), JSON (model_dump), Markdown
+- **API:** `query()` pipeline — acquire, parse, chunk, index, search, assemble
+- **CLI:** `archex query`, `archex cache list|clean|info`
+- 174 tests, 85% coverage
+
+### Phase 3 — Intelligence
+
+- **Modules:** Louvain community detection on file-level dependency graph
+- **Patterns:** Rule-based detection — middleware chain, plugin system, event bus, repository/DAO, strategy
+- **Interfaces:** Public API surface extraction with usage counts
+- **TypeScript adapter:** ES modules, CommonJS, type-only imports, re-exports, index resolution
+- **LLM enrichment:** Optional `Provider` protocol (Anthropic, OpenAI, OpenRouter), structured output
+- **Decisions:** `infer_decisions()` with structural evidence + optional LLM inference
+- Full `ArchProfile` assembly with modules, patterns, interfaces, decisions
+
+### Phase 4 — Compare + Polish
+
+- **Go adapter:** Functions, methods (pointer/value receivers), structs, interfaces, const/var, Go visibility (uppercase = public)
+- **Rust adapter:** fn, struct, enum, trait, impl blocks, const/static, macro_rules, pub/pub(crate)/pub(super) visibility
+- **Vector index:** Numpy-based cosine similarity (L2-norm at build, dot at search), `.npz` persistence
+- **Embedder protocol:** `encode(texts) -> list[list[float]]`, `dimension -> int` — Nomic Code ONNX, API (OpenAI-compatible), SentenceTransformers backends
+- **Hybrid retrieval:** Reciprocal rank fusion merging BM25 + vector results by chunk ID
+- **Comparison engine:** `compare_repos()` across 6 structural dimensions (api_surface, concurrency, configuration, error_handling, state_management, testing), no LLM required
+- **CLI polish:** `--timing` flag on analyze/query, `--strategy bm25|hybrid` on query, `--dimensions` on compare
+- 372 tests, 81% coverage

archex-0.4.0/PKG-INFO

@@ -0,0 +1,301 @@
+Metadata-Version: 2.4
+Name: archex
+Version: 0.4.0
+Summary: Architecture extraction & codebase intelligence for the agentic era
+Author: Tom
+License: MIT
+License-File: LICENSE
+Keywords: architecture,ast,code-intelligence,codebase,rag,retrieval
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.11
+Requires-Dist: click>=8.1
+Requires-Dist: networkx>=3.3
+Requires-Dist: pydantic>=2.7
+Requires-Dist: tiktoken>=0.7
+Requires-Dist: tree-sitter-go>=0.23
+Requires-Dist: tree-sitter-javascript>=0.23
+Requires-Dist: tree-sitter-python>=0.23
+Requires-Dist: tree-sitter-rust>=0.23
+Requires-Dist: tree-sitter-typescript>=0.23
+Requires-Dist: tree-sitter>=0.23
+Provides-Extra: all
+Requires-Dist: anthropic>=0.30; extra == 'all'
+Requires-Dist: langchain-core>=0.2; extra == 'all'
+Requires-Dist: llama-index-core>=0.10; extra == 'all'
+Requires-Dist: mcp>=1.0; extra == 'all'
+Requires-Dist: onnxruntime>=1.17; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Requires-Dist: tokenizers>=0.15; extra == 'all'
+Requires-Dist: voyageai>=0.3; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.30; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: pre-commit>=3.7; extra == 'dev'
+Requires-Dist: pyright>=1.1; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=0.2; extra == 'langchain'
+Provides-Extra: llamaindex
+Requires-Dist: llama-index-core>=0.10; extra == 'llamaindex'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0; extra == 'mcp'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Provides-Extra: vector
+Requires-Dist: onnxruntime>=1.17; extra == 'vector'
+Requires-Dist: tokenizers>=0.15; extra == 'vector'
+Provides-Extra: vector-torch
+Requires-Dist: sentence-transformers>=2.6; extra == 'vector-torch'
+Provides-Extra: voyage
+Requires-Dist: voyageai>=0.3; extra == 'voyage'
+Description-Content-Type: text/markdown
+
+# archex
+
+Architecture extraction and codebase intelligence for the agentic era.
+
+archex is a Python library and CLI that transforms any Git repository into structured architectural intelligence and token-budget-aware code context. It serves two consumers from a single index: **human architects** receive an `ArchProfile` with module boundaries, dependency graphs, detected patterns, and interface surfaces; **AI agents** receive a `ContextBundle` with relevance-ranked, syntax-aligned code chunks assembled to fit within a specified token budget.
+
+## Features
+
+- **4 language adapters** — Python, TypeScript/JavaScript, Go, Rust (tree-sitter AST parsing), extensible via entry points
+- **3 public APIs** — `analyze()`, `query()`, `compare()`
+- **Hybrid retrieval** — BM25 keyword search + optional vector embeddings with reciprocal rank fusion
+- **Token budget assembly** — AST-aware chunking, dependency-graph expansion, greedy bin-packing with configurable `ScoringWeights`
+- **Structural analysis** — module detection (Louvain), pattern recognition (extensible `PatternRegistry`), interface extraction
+- **Cross-repo comparison** — 6 architectural dimensions, no LLM required
+- **Performance** — cache-first query (skips parse on cache hit), parallel parsing, parallel compare, git-aware cache keys
+- **Extensibility** — plugin APIs for language adapters, pattern detectors, chunkers, and scoring weights via entry points and protocols
+- **Security** — input validation on git URLs/branches, FTS5 query escaping, cache key validation, `allow_pickle=False` for vector persistence
+- **LLM-optional** — entire structural pipeline runs without API calls; LLM enrichment is opt-in
+
+## Installation
+
+```bash
+uv add archex
+```
+
+### Extras
+
+| Extra                  | What it adds                             |
+| ---------------------- | ---------------------------------------- |
+| `archex[vector]`       | ONNX-based local embeddings (Nomic Code) |
+| `archex[vector-torch]` | Torch-backed sentence-transformers       |
+| `archex[voyage]`       | Voyage Code API embeddings               |
+| `archex[openai]`       | OpenAI API embeddings + LLM enrichment   |
+| `archex[anthropic]`    | Anthropic API LLM enrichment             |
+| `archex[mcp]`          | MCP server for agent integration         |
+| `archex[langchain]`    | LangChain retriever integration          |
+| `archex[llamaindex]`   | LlamaIndex retriever integration         |
+| `archex[all]`          | All optional dependencies                |
+
+## Quick Start
+
+### Python API
+
+```python
+from archex import analyze, query, compare
+from archex.models import RepoSource
+
+# Architectural analysis
+profile = analyze(RepoSource(local_path="./my-project"))
+for module in profile.module_map:
+    print(f"{module.name}: {len(module.files)} files")
+for pattern in profile.pattern_catalog:
+    print(f"[{pattern.confidence:.0%}] {pattern.name}")
+
+# Implementation context for an agent
+bundle = query(
+    RepoSource(local_path="./my-project"),
+    "How does authentication work?",
+    token_budget=8192,
+)
+print(bundle.to_prompt(format="xml"))
+
+# Query with custom scoring weights
+from archex.models import ScoringWeights
+
+bundle = query(
+    RepoSource(local_path="./my-project"),
+    "database connection pooling",
+    scoring_weights=ScoringWeights(relevance=0.8, structural=0.1, type_coverage=0.1),
+)
+
+# Cross-repo comparison
+result = compare(
+    RepoSource(local_path="./project-a"),
+    RepoSource(local_path="./project-b"),
+    dimensions=["error_handling", "api_surface"],
+)
+```
+
+### CLI
+
+```bash
+# Analyze a local repo or remote URL
+archex analyze ./my-project --format json
+archex analyze https://github.com/org/repo --format markdown -l python --timing
+
+# Query for implementation context
+archex query ./my-project "How does auth work?" --budget 8192 --format xml
+archex query ./my-project "connection pooling" --strategy hybrid --timing
+
+# Compare two repositories
+archex compare ./project-a ./project-b --dimensions error_handling,api_surface --format markdown
+
+# Manage the analysis cache
+archex cache list
+archex cache clean --max-age 168
+archex cache info
+```
+
+## CLI Reference
+
+### `archex analyze <source>`
+
+Analyze a repository and produce an architecture profile.
+
+| Option              | Default | Description                       |
+| ------------------- | ------- | --------------------------------- |
+| `--format`          | `json`  | Output format: `json`, `markdown` |
+| `-l` / `--language` | all     | Filter by language (repeatable)   |
+| `--timing`          | off     | Print timing breakdown to stderr  |
+
+### `archex query <source> <question>`
+
+Query a repository and return a context bundle.
+
+| Option              | Default | Description                              |
+| ------------------- | ------- | ---------------------------------------- |
+| `--budget`          | `8192`  | Token budget for context assembly        |
+| `--format`          | `xml`   | Output format: `xml`, `json`, `markdown` |
+| `-l` / `--language` | all     | Filter by language (repeatable)          |
+| `--strategy`        | `bm25`  | Retrieval strategy: `bm25`, `hybrid`     |
+| `--timing`          | off     | Print timing breakdown to stderr         |
+
+### `archex compare <source_a> <source_b>`
+
+Compare two repositories across architectural dimensions.
+
+| Option              | Default | Description                       |
+| ------------------- | ------- | --------------------------------- |
+| `--dimensions`      | all 6   | Comma-separated dimension list    |
+| `--format`          | `json`  | Output format: `json`, `markdown` |
+| `-l` / `--language` | all     | Filter by language (repeatable)   |
+| `--timing`          | off     | Print timing breakdown to stderr  |
+
+Supported dimensions: `api_surface`, `concurrency`, `configuration`, `error_handling`, `state_management`, `testing`.
+
+### `archex mcp`
+
+Start the MCP (Model Context Protocol) server for agent integration.
+
+Tools exposed: `analyze_repo`, `query_repo`, `compare_repos`.
+
+### `archex cache <subcommand>`
+
+Manage the local analysis cache.
+
+| Subcommand | Options                              | Description            |
+| ---------- | ------------------------------------ | ---------------------- |
+| `list`     | `--cache-dir`                        | List cached entries    |
+| `clean`    | `--max-age N` (hours), `--cache-dir` | Remove expired entries |
+| `info`     | `--cache-dir`                        | Show cache summary     |
+
+## Extensibility
+
+archex supports plugin APIs via Python entry points and protocols.
+
+### Custom Language Adapters
+
+Register a language adapter in your package's `pyproject.toml`:
+
+```toml
+[project.entry-points."archex.language_adapters"]
+java = "mypackage.adapters:JavaAdapter"
+```
+
+The adapter class must implement the `LanguageAdapter` protocol (see `archex.parse.adapters.base`).
+
+### Custom Pattern Detectors
+
+Register a pattern detector function:
+
+```toml
+[project.entry-points."archex.pattern_detectors"]
+my_pattern = "mypackage.patterns:detect_my_pattern"
+```
+
+Detector signature: `(list[ParsedFile], DependencyGraph) -> DetectedPattern | None`.
+
+### Custom Chunkers
+
+Implement the `Chunker` protocol and pass it to `query()`:
+
+```python
+from archex.index.chunker import Chunker
+
+class MyChunker:
+    def chunk_file(self, parsed_file, source): ...
+    def chunk_files(self, parsed_files, sources): ...
+
+bundle = query(source, question, chunker=MyChunker())
+```
+
+### Custom Scoring Weights
+
+Adjust the retrieval ranking formula:
+
+```python
+from archex.models import ScoringWeights
+
+# Boost relevance, lower structural weight
+weights = ScoringWeights(relevance=0.8, structural=0.1, type_coverage=0.1)
+bundle = query(source, question, scoring_weights=weights)
+```
+
+## Configuration
+
+archex reads configuration from `~/.archex/config.toml` and `ARCHEX_*` environment variables.
+
+```toml
+# ~/.archex/config.toml
+[default]
+languages = ["python", "typescript"]
+cache = true
+cache_dir = "~/.archex/cache"
+parallel = true
+max_file_size = 10000000
+```
+
+Environment variables override file config: `ARCHEX_CACHE_DIR`, `ARCHEX_PARALLEL`, `ARCHEX_MAX_FILE_SIZE`.
+
+## Development
+
+```bash
+git clone https://github.com/Mathews-Tom/archex.git
+cd archex
+uv sync --all-extras
+
+# Run tests (538 tests)
+uv run pytest
+
+# Lint and format
+uv run ruff check .
+uv run ruff format .
+
+# Type check (strict mode)
+uv run pyright
+```
+
+## License
+
+Apache 2.0 — see [LICENSE](LICENSE).