adl-toolkit 0.0.0-stage → 0.1.1

package/INSTALL_TOOLKIT.md

@@ -0,0 +1,163 @@
+# ADL Toolkit — User Guide
+
+ADL Toolkit connects a code graph tool to your AI assistants. Choose ADL-Drift,
+CodeGraph, Git Nexus, or Graphify, and optionally add Caveman and Ponytail.
+
+Use two commands to manage your setup:
+
+```bash
+npx adl-toolkit install
+npx adl-toolkit uninstall
+```
+
+Follow the steps below to install the toolkit, start using it, and remove it when needed.
+
+## First-time checklist
+
+Before you start, install:
+
+- Node.js **22.20+ within the 22.x series, or 24.11+**.
+- npm and npx, included with Node.js.
+- Python **3.11+** and Git.
+- At least one supported AI assistant: Codex, Claude Code, Cursor, or Antigravity.
+
+Open a terminal and check:
+
+```bash
+node --version
+npm --version
+git --version
+```
+
+Check Python with `python3 --version` on Linux, macOS, or WSL, or
+`py -3 --version` / `python --version` on Windows.
+On Windows, use `npm.cmd` and `npx.cmd` if PowerShell blocks npm or npx scripts.
+After installing or upgrading a prerequisite, reopen your terminal.
+
+## 1. Install
+
+Open a terminal in the project you want your AI assistant to explore, then run:
+
+```bash
+npx adl-toolkit install
+```
+
+If npm asks to download `adl-toolkit`, confirm to continue.
+
+### Linux and macOS
+
+```bash
+cd /path/to/my-project
+npx adl-toolkit install
+```
+
+### Running on Windows (PowerShell or CMD)
+
+Use Windows Node.js and Python:
+
+```powershell
+cd "C:\work\my-project"
+npx.cmd adl-toolkit install
+```
+
+In CMD, use `cd /d` when changing drives.
+
+### Running from WSL
+
+Use Linux Node.js, npm, and Python inside WSL when your AI assistant also runs there:
+
+```bash
+cd /mnt/e/work/my-project
+npx adl-toolkit install
+```
+
+If your assistant runs on Windows, follow the Windows instructions instead.
+For mixed Windows/Linux tool errors, see [WSL recovery](windows_installation.md#recovery-in-wsl).
+
+### Choose your setup
+
+The installer walks you through these choices:
+
+1. **Graph tool:** select ADL-Drift, CodeGraph, Git Nexus, or Graphify.
+2. **Optional plugins:** select Caveman for concise agent responses, Ponytail for minimal coding changes, both, or neither.
+3. **AI assistants:** select one or more assistants, or choose **Select all**.
+4. **Project directory:** confirm the project you want to analyze.
+5. **Graph index:** keep **Yes (recommended)** to build the index during installation. Large projects may take longer.
+6. **Review:** check your selections and confirm **Install now?**.
+
+Use **↑/↓** to move, **Space** to toggle selections, **Enter** to save and continue,
+**←** to return to the previous step, and **Ctrl+C** to cancel.
+
+Wait for **Installation complete.** Follow any next steps printed by the installer.
+Reopen your terminal if it reports PATH changes, then restart your selected AI assistants.
+
+## 2. Use the toolkit
+
+Open the same project in one of the assistants you selected during installation.
+Start a new conversation and ask it to use your selected graph tool. For example:
+
+> Use the code graph to explain this project's main modules and entry points.
+
+> Find the callers of this function and show which parts of the project a change could affect.
+
+If you skipped indexing, run the graph build command printed after installation first.
+To refresh the index after changing your code, use that tool's graph build command again.
+
+Selected Caveman and Ponytail plugins are activated through the project's agent instructions.
+To disable one, ask your assistant not to use it, or remove it through the uninstall menu.
+
+## 3. Uninstall
+
+Open a terminal in the project where you installed the toolkit:
+
+```bash
+npx adl-toolkit uninstall
+```
+
+On Windows, use `npx.cmd adl-toolkit uninstall`.
+
+1. Confirm the **Project directory**.
+2. Choose **Entire managed installation**, or remove selected plugins or AI assistant connections when those options are available.
+3. For a full uninstall, choose whether to remove generated index data. The default **No** keeps it.
+4. Review the summary and select **Remove** to confirm. **Cancel** is selected by default.
+
+Removing selected plugins or assistant connections keeps the shared graph runtime.
+To switch graph tools, uninstall the current managed installation, then run
+`npx adl-toolkit install` and select the new tool.
+
+If uninstall reports that a file was changed and retained, review the reported file
+before retrying. If it cannot find your installation, check that you selected the original project directory.
+
+## Everyday commands
+
+Install and uninstall are the main commands. For troubleshooting or maintenance,
+run these advanced commands from your project directory:
+
+```bash
+# Inspect installation files
+npx adl-toolkit status
+
+# Check installation health, including ADL-Drift's live MCP connection
+npx adl-toolkit doctor
+
+# Restore missing managed configuration and instructions
+npx adl-toolkit repair
+
+# Upgrade the managed graph runtime
+npx adl-toolkit update
+
+# Show all commands and options
+npx adl-toolkit --help
+```
+
+Update upgrades the runtime; it does not replace the graph build command used to refresh your index.
+On Windows, use `npx.cmd` for these commands too.
+
+## If something fails
+
+- **npm or npx is missing:** install or repair Node.js, then reopen your terminal.
+- **Python cannot be found:** check its version and PATH. If needed, set `ADL_TOOLKIT_PYTHON` to your Python executable path.
+- **An installation step fails:** open the full audit log at the path printed by the installer, then retry the reported command.
+- **Your assistant cannot connect:** restart it in the installed project and run `npx adl-toolkit doctor`. For ADL-Drift, keep the failing checks and server traceback when reporting the issue.
+- **Configuration is missing:** run `npx adl-toolkit repair`, then restart your assistant.
+- **Windows or WSL cannot launch the command:** see [Windows recovery](windows_installation.md#recovery-in-native-windows) or [WSL recovery](windows_installation.md#recovery-in-wsl).

package/README.md

@@ -1,3 +1,270 @@
-# Temporary Holding Version
+# ADL Drift
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+ADL Drift indexes and analyzes Python and Java source code with a pure-Python runtime.
+
+## Quick Start
+
+**Connect a graph tool to your AI assistants with ADL Toolkit.**
+Check the [prerequisites](INSTALL_TOOLKIT.md#first-time-checklist),
+then open a terminal in your project and run:
+
+```bash
+npx adl-toolkit install
+```
+
+Choose your graph tool, optional plugins, and AI assistants in the menu.
+Confirm your project, build the index, then restart your assistants to start using the graph.
+
+To remove the setup or selected components, run:
+
+```bash
+npx adl-toolkit uninstall
+```
+
+On Windows, use `npx.cmd`. Follow the [ADL Toolkit user guide](INSTALL_TOOLKIT.md)
+for the full installation, usage, and uninstall walkthrough, including
+[advanced commands](INSTALL_TOOLKIT.md#everyday-commands) and troubleshooting.
+
+**Using ADL Drift from an AI agent (Claude Code / Cursor / Antigravity / Codex VS
+Code)?**
+
+For Claude, Cursor, or Antigravity:
+
+```bash
+pip install -e . && adl-drift mcp setup
+```
+
+Then verify with `adl-drift mcp doctor`. Full guide (harness vs MCP-only,
+CI gate, watch mode, troubleshooting): **[SETUP.md](SETUP.md)**.
+
+Codex VS Code is included: setup writes its trusted-project
+`.codex/config.toml` entry. Use the full [VS Code demo](DEMO_HAPPYCASE_VSCODE.md)
+for a walkthrough.
+
+For a Java/JUnit 5 indexing demo, including the reproducible cold-scan and
+incremental-correctness commands, see [DEMO_JAVA_JUNIT.md](DEMO_JAVA_JUNIT.md).
+
+```bash
+# 1. Scan the project — this now also builds and persists the graph
+adl-drift -p adl_drift scan adl_drift
+
+# 2. Run graph stats (first time builds + stores)
+adl-drift -p adl_drift graph
+
+# 3. Run it again — should be instant because it loads from SQLite
+adl-drift -p adl_drift graph
+
+# 4. Impact queries are also cached
+adl-drift -p adl_drift impact add_knowledge
+```
+
+## Command Reference
+
+> Full plain-language reference with every option, output-column explanations, and workflows: **[COMMANDS.md](COMMANDS.md)**.
+
+Global option (goes **before** the command): `--project, -p <dir>` — project root (default `.`). Example: `adl-drift -p adl_drift scan adl_drift`.
+
+| Command | What it does | Options |
+|---------|--------------|---------|
+| `adl-drift scan [PATH]` | Scan files/directories for patterns; also builds + persists the call graph | `-c, --category <name>` filter to one category (e.g. `security`) · `-f, --format table\|json\|github` · `-i, --incremental` only re-parse changed files · `--more-info` beginner-friendly output (see below) · `--full` show all rows with `--more-info` |
+| `adl-drift patterns` | List every stored pattern with its ID and approval state | — |
+| `adl-drift approve <id>` | Mark a pattern as approved ("this is intentional, stop flagging it") | — |
+| `adl-drift gate` | CI gate — exits 1 if unapproved patterns exist | `--json` machine contract: `{violations, new_patterns, scan_errors, high_security}`; exits 1 only on HIGH security findings or scan errors |
+| `adl-drift graph` | Call-graph statistics: node/edge counts by type, entry points | — |
+| `adl-drift impact <symbol>` | Blast radius: who calls `<symbol>`, directly and transitively. Accepts qualified (`pkg.mod.Class.method`) or simple names | `-f, --format tree\|table\|json` (default `tree`) · `--include-initializers` counts Java `<init>`/`<clinit>` contexts in caller totals |
+| `adl-drift viz` | Interactive graph visualization in the browser (live-updates on file save) | `--host` (default `127.0.0.1`) · `--port` (default `8765`) |
+| `adl-drift learn <topic> [text...]` | Store tribal knowledge (no text → opens `$EDITOR`, or pipe via stdin) | — |
+| `adl-drift why <topic>` | Retrieve tribal knowledge for a topic | — |
+| `adl-drift server` | MCP server over HTTP/SSE | `--port` (default `3000`) |
+| `adl-drift mcp serve` | MCP server over stdio (Claude Code / Cursor / Claude Desktop / Antigravity / Codex VS Code) | — |
+| `adl-drift mcp setup` | Auto-configure MCP clients: writes `.mcp.json` / `.cursor/mcp.json` / `.codex/config.toml` / `~/.gemini/config/mcp_config.json` + installs skill files (merge, never clobber) | `--claude` · `--cursor` · `--antigravity` · `--codex` · `--print` dry run |
+| `adl-drift mcp doctor` | Verify wiring: config → interpreter → SDK → live stdio handshake → tool list | — |
+| `adl-drift watch` | Keep the index fresh: hash-based polling + incremental rescans | `--interval <s>` (default 2) |
+
+### `scan --more-info` (easy mode)
+
+Beginner-friendly report: summary line (`Scanned 59 files · 847 findings · 4 security (3 HIGH, 1 MEDIUM)`), per-category counts, a plain-language **What was found** column, color-coded **Severity**, confidence as words, and the first 50 rows with a footer hint (`--full` lifts the cap).
+
+```bash
+adl-drift -p . scan . --more-info
+adl-drift -p . scan . --more-info --full          # all rows
+adl-drift -p . scan . --more-info -c security     # one category
+```
+
+## Reference: Categories, Types & Values
+
+### Pattern categories (the `Category` column in scan output)
+
+| Category | What it covers |
+|----------|----------------|
+| `ROUTES` | Web endpoints from Flask/FastAPI/Django and Spring MVC/JAX-RS |
+| `ERRORS` | Python error handling: broad/empty handlers and custom exception classes |
+| `DATA_ACCESS` | Python ORM/raw SQL plus JDBC, JPA/Hibernate, Spring Data, and MyBatis operations |
+| `DECORATORS` | Python decorators and Java annotations used for framework behavior |
+| `ASYNC` | Python async/await patterns and Java asynchronous APIs |
+| `IMPORTS` | Python and Java imports with language-appropriate classification |
+| `SECURITY` | Risky execution, SQL construction, deserialization, crypto, reflection, secrets, and production assertions — carries a **Severity** |
+| `TESTS` | pytest/unittest and JUnit/TestNG tests, fixtures, and lifecycle methods |
+| `CONFIG` | Environment/config reads and constant conventions |
+| `LOGGING` | Console and logger calls, including inefficient message construction |
+| `TYPES` | Python annotation coverage and Java raw/nullability type conventions |
+
+### Scan table columns
+
+| Column | Meaning |
+|--------|---------|
+| `Object` | The concrete thing the finding is about — the function, decorator, imported module, env-var key, etc. Empty when the finding has no single symbol (e.g. a bare `except:`) |
+| `Severity` | Security/logging findings only: `HIGH` (red — fix or justify), `MEDIUM` (yellow — review), `LOW` (dim — informational) |
+| `Confidence` | How sure the detector is. Numeric `0–1` in the default table; words in `--more-info`: **Certain** ≥ 0.85, **Likely** ≥ 0.6, **Guess** below |
+| `Line` | 1-indexed line in the file |
+
+### Graph node types (`adl-drift graph`, viz UI)
+
+| Node type | Meaning | Viz shape |
+|-----------|---------|-----------|
+| `MODULE` | One `.py` or `.java` file (`pkg.sub.mod`) | rounded box, dark |
+| `CLASS` | Class definition | diamond, grey |
+| `FUNCTION` | Function or method | dot, emerald |
+| `VARIABLE` | Module-level constant or `self.X` attribute defined in `__init__` | dot |
+
+Node IDs are dotted qualified names — `module.Class.method` — relative to the detected project root.
+
+### Graph edge types
+
+| Edge | Meaning | Emitted confidence |
+|------|---------|--------------------|
+| `CONTAINS` | Module contains class/function; class contains methods | 1.0 |
+| `CALLS` | Function A calls function B | 0.3–0.95 by resolution strategy (see below) |
+| `AWAITS` | The call is awaited (`await f()`) — coexists with `CALLS` | same as the call |
+| `IMPORTS` | Module imports another module | 1.0 |
+| `INHERITS` | Class extends a base class | 1.0 |
+| `DECORATED_BY` | Symbol wrapped by a decorator function | 1.0 |
+| `REFERENCES` | Function reads a variable/constant (`return MAX_RETRIES`, `self.db`) | 0.85–0.95 |
+| `TYPE_OF` | Function has a parameter annotated with a class (`def f(s: Service)`) | 1.0 |
+| `RETURNS` | Function's return annotation is a class (`-> Service`) | 1.0 |
+
+`CALLS` confidence by how the call was resolved: `0.95` `self.x()` on own class · `0.9` same-module name · `0.85` via import / inherited method / `super()` / type-annotated receiver · `0.75` chained call (`lib.Db.query()`) · `0.6` unique global name · `0.3–0.5` guessed by attribute name. The viz draws ≥ 0.85 solid, 0.6–0.85 dashed, < 0.6 faint.
+
+**Blast radius** (`adl-drift impact`) follows incoming `CALLS`, `REFERENCES`, `TYPE_OF`, and `RETURNS` edges — so changing a constant reports every function that reads it, and changing a class reports every function annotated with it, plus their callers transitively. Synthetic Java field/initializer-block contexts (`<init>`/`<clinit>`) are reported in separate initializer fields by default; pass `--include-initializers` to include them in the ordinary caller totals.
+
+### Symbol kinds (internal `Symbol.kind`)
+
+`function` (incl. methods — enclosing class recorded in `parent`) · `class` · `variable` (module-level assignment) · `attribute` (`self.X = ...` in `__init__`).
+
+## Node ID Collision & Project Root Detection
+
+In large codebases and monorepos, multiple packages can contain files with the same relative path (e.g. `backend/api/users.py` and `frontend/api/users.py`). ADL Drift automatically detects your project root by looking for `pyproject.toml`, `.git`, `setup.py`, or `setup.cfg` and computes fully-qualified module names relative to that root. This prevents symbols from overwriting each other in the call graph.
+
+### How it works
+
+- **Detection**: `CallGraphBuilder.find_project_root()` walks up the directory tree from the scan target until it finds a root marker.
+- **Module naming**: `_path_to_module()` strips the project root prefix from every file path, so `/repo/backend/api/users.py` becomes `backend.api.users` instead of a truncated heuristic.
+- **Fallback**: If no project root is found, ADL Drift falls back to the last 3 path components for backward compatibility.
+
+### Verifying the fix
+
+Run the collision test to see the before/after behavior:
+
+```bash
+pytest tests/test_graph.py::TestCallGraphBuilder::test_no_node_id_collision_with_project_root -v
+```
+
+Or inspect the graph in a real project and confirm that nested packages produce distinct node IDs:
+
+```bash
+adl-drift -p . graph
+```
+
+### Test scripts
+
+These are the exact scripts used during development to verify the collision fix.
+
+**1. Full collision demo (shows old vs new behavior side-by-side)**
+
+```python
+from pathlib import Path
+import tempfile
+from adl_drift.parser.core import PythonParser
+from adl_drift.parser.extractor import SymbolExtractor
+from adl_drift.graph.builder import CallGraphBuilder
+
+with tempfile.TemporaryDirectory() as td:
+    root = Path(td)
+    # Paths where the last 3 components are IDENTICAL — old code collides here
+    (root / 'services' / 'backend' / 'api' / 'v1').mkdir(parents=True)
+    (root / 'services' / 'frontend' / 'api' / 'v1').mkdir(parents=True)
+    (root / 'pyproject.toml').write_text('[project]\n')
+
+    (root / 'services' / 'backend' / 'api' / 'v1' / 'users.py').write_text('def list(): pass\n')
+    (root / 'services' / 'frontend' / 'api' / 'v1' / 'users.py').write_text('def list(): pass\n')
+
+    parser = PythonParser()
+    extractor = SymbolExtractor(parser)
+    r1 = extractor.extract(root / 'services' / 'backend' / 'api' / 'v1' / 'users.py')
+    r2 = extractor.extract(root / 'services' / 'frontend' / 'api' / 'v1' / 'users.py')
+
+    # OLD: last-3 heuristic collapses both files into api.v1.users
+    old = CallGraphBuilder(project_root=None)
+    old.add_file_result(r1)
+    old.add_file_result(r2)
+    print('OLD nodes:', sorted(old._nodes))
+    print('OLD count :', len(old._nodes), '(should be 4, is 2 -> COLLISION)')
+
+    # NEW: full relative path from project root
+    new = CallGraphBuilder(project_root=root)
+    new.add_file_result(r1)
+    new.add_file_result(r2)
+    print('NEW nodes:', sorted(new._nodes))
+    print('NEW count :', len(new._nodes), '(4 -> DISTINCT)')
+```
+
+Expected output:
+
+```
+OLD nodes: ['api.v1.users', 'api.v1.users.list']
+OLD count : 2 (should be 4, is 2 -> COLLISION)
+NEW nodes: ['services.backend.api.v1.users', 'services.backend.api.v1.users.list',
+            'services.frontend.api.v1.users', 'services.frontend.api.v1.users.list']
+NEW count : 4 (4 -> DISTINCT)
+```
+
+**2. Quick live-project check**
+
+```python
+from pathlib import Path
+from adl_drift.cli.commands import DriftCommands
+
+cmd = DriftCommands(Path('.'))
+stats = cmd.graph_stats()
+print('Nodes by type:', stats.get('node_types'))
+print('Total nodes  :', stats.get('total_nodes'))
+```
+
+**3. Unit test (also in `tests/test_graph.py`)**
+
+```python
+def test_no_node_id_collision_with_project_root(self, tmp_path):
+    """P0.2: Files with same relative path in different subtrees must have unique node IDs."""
+    backend = tmp_path / "backend" / "api"
+    frontend = tmp_path / "frontend" / "api"
+    backend.mkdir(parents=True)
+    frontend.mkdir(parents=True)
+
+    (backend / "users.py").write_text("def get_users(): pass\n")
+    (frontend / "users.py").write_text("def get_users(): pass\n")
+
+    (tmp_path / "pyproject.toml").write_text("[project]\n")
+
+    extractor = SymbolExtractor(PythonParser())
+    result1 = extractor.extract(backend / "users.py")
+    result2 = extractor.extract(frontend / "users.py")
+
+    builder = CallGraphBuilder(project_root=tmp_path)
+    builder.add_file_result(result1)
+    builder.add_file_result(result2)
+
+    assert "backend.api.users" in builder._nodes
+    assert "frontend.api.users" in builder._nodes
+    assert "backend.api.users.get_users" in builder._nodes
+    assert "frontend.api.users.get_users" in builder._nodes
+```

package/adl_drift/__init__.py

@@ -0,0 +1,6 @@
+"""
+ADL Drift - Static Analysis Engine
+A Python reimplementation of the Drift v2 architecture using tree-sitter.
+"""
+
+__version__ = "0.1.0"

package/adl_drift/cli/__init__.py

@@ -0,0 +1 @@
+"""Command-line interface."""