archeologist 0.1.0__tar.gz

archeologist-0.1.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: archeologist
+Version: 0.1.0
+Summary: Code archaeologist - reconstruct function decision history via AST-aware lineage tracking
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: GitPython
+Requires-Dist: PyGithub
+Requires-Dist: tree-sitter>=0.21.0
+Requires-Dist: tree-sitter-languages>=1.9.0
+Requires-Dist: click
+Requires-Dist: pytest
+Requires-Dist: python-dotenv
+Requires-Dist: pyyaml
+
+# Archeologist - Semantic Lineage Graph Generator
+
+Post-incident archaeology tool that reconstructs function decision history via deterministic AST-aware lineage tracking.
+
+## CLI Usage
+
+```bash
+# Install
+pip install -e .
+
+# Analyze file (auto-detects git repo)
+arc analyze path/to/file.py
+
+# Analyze specific function  
+arc analyze-function path/to/file.py function_name
+
+# With GitHub PR integration
+arc analyze-function path/to/file.py function_name --repo owner/repo
+
+# With LLM narrative synthesis
+export CLAUDE_API_KEY=sk-ant-xxx
+arc analyze-function path/to/file.py function_name
+```
+
+## Configuration
+
+Set environment variables:
+
+```bash
+export GITHUB_TOKEN=ghp_xxx
+export CLAUDE_API_KEY=sk-ant-xxx
+export GIT_REPO_PATH=/path/to/local/repo
+```
+
+## Architecture
+
+Three-phase pipeline:
+
+1. **Semantic Lineage Tracking**
+   - GitWalker traverses history (--no-renames flag)
+   - ASTParser extracts function boundaries (Python, JS, TS, Go, Rust, Java, C, C++, Ruby, PHP)
+   - LineageTracker links nodes via four-tier hierarchy
+
+2. **Contextual Slicing**
+   - PRFetcher pulls associated PRs
+   - Geographic filter maps review comments to AST node line ranges
+
+3. **Narrative Synthesis**
+   - LiteLLM abstracts LLM calls (Claude, local models)
+   - Outputs 5-sentence brief explaining decisions
+
+## MCP Server
+
+The tool exposes an MCP-compatible JSON-RPC 2.0 server over stdio:
+
+```bash
+# Run MCP server
+arc-mcp
+
+# Or run directly
+python -m src.mcp.server
+```
+
+### Available Methods
+
+```json
+// List functions in a file
+{"jsonrpc": "2.0", "id": 1, "method": "list_functions", "params": {"file_path": "/path/to/file.py"}}
+
+// Analyze a specific function
+{"jsonrpc": "2.0", "id": 2, "method": "analyze_function", "params": {"file_path": "/path/to/file.py", "function_name": "foo"}}
+
+// Analyze a file's overall lineage
+{"jsonrpc": "2.0", "id": 3, "method": "analyze_file", "params": {"file_path": "/path/to/file.py"}}
+```
+
+## Example
+
+```bash
+# Analyze the `authenticate` function in a FastAPI project
+GIT_REPO_PATH=/Users/fuads/fastapi arc analyze-function app/auth.py authenticate --repo fastapi/fastapi
+```
+
+Output:
+```
+Analyzing function authenticate in app/auth.py...
+Found 12 lineage edges for authenticate
+Summary: Found 12 historical versions of this code. Change types: physical: 8, identity: 4
+```
+
+With LLM synthesis:
+```
+The authenticate function evolved through 12 commits over 18 months. 
+Initial implementation used simple token validation, replaced in PR #2341 
+with OAuth2 Bearer token parsing after security audit. Several performance 
+optimizations were attempted (PRs #1892, #2103) but reverted due to race 
+conditions. The current implementation handles both JWT and opaque tokens 
+with a unified interface, consolidating three previous approaches.
+```
+
+## Testing
+
+```bash
+pytest tests/
+```
+
+## Roadmap
+
+- [x] Graph construction (GitWalker, ASTParser, LineageTracker)
+- [x] Contextual slicing (PRFetcher, GeographicFilter)
+- [x] CLI commands with local git repo auto-detection
+- [x] 10 language support (Python, JS, TS, Go, Rust, Java, C, C++, Ruby, PHP)
+- [x] Real-world testing on Flask repo
+- [x] MCP server (JSON-RPC 2.0 over stdio, works with Python 3.9)
+
+## License
+
+MIT

archeologist-0.1.0/README.md

@@ -0,0 +1,118 @@
+# Archeologist - Semantic Lineage Graph Generator
+
+Post-incident archaeology tool that reconstructs function decision history via deterministic AST-aware lineage tracking.
+
+## CLI Usage
+
+```bash
+# Install
+pip install -e .
+
+# Analyze file (auto-detects git repo)
+arc analyze path/to/file.py
+
+# Analyze specific function  
+arc analyze-function path/to/file.py function_name
+
+# With GitHub PR integration
+arc analyze-function path/to/file.py function_name --repo owner/repo
+
+# With LLM narrative synthesis
+export CLAUDE_API_KEY=sk-ant-xxx
+arc analyze-function path/to/file.py function_name
+```
+
+## Configuration
+
+Set environment variables:
+
+```bash
+export GITHUB_TOKEN=ghp_xxx
+export CLAUDE_API_KEY=sk-ant-xxx
+export GIT_REPO_PATH=/path/to/local/repo
+```
+
+## Architecture
+
+Three-phase pipeline:
+
+1. **Semantic Lineage Tracking**
+   - GitWalker traverses history (--no-renames flag)
+   - ASTParser extracts function boundaries (Python, JS, TS, Go, Rust, Java, C, C++, Ruby, PHP)
+   - LineageTracker links nodes via four-tier hierarchy
+
+2. **Contextual Slicing**
+   - PRFetcher pulls associated PRs
+   - Geographic filter maps review comments to AST node line ranges
+
+3. **Narrative Synthesis**
+   - LiteLLM abstracts LLM calls (Claude, local models)
+   - Outputs 5-sentence brief explaining decisions
+
+## MCP Server
+
+The tool exposes an MCP-compatible JSON-RPC 2.0 server over stdio:
+
+```bash
+# Run MCP server
+arc-mcp
+
+# Or run directly
+python -m src.mcp.server
+```
+
+### Available Methods
+
+```json
+// List functions in a file
+{"jsonrpc": "2.0", "id": 1, "method": "list_functions", "params": {"file_path": "/path/to/file.py"}}
+
+// Analyze a specific function
+{"jsonrpc": "2.0", "id": 2, "method": "analyze_function", "params": {"file_path": "/path/to/file.py", "function_name": "foo"}}
+
+// Analyze a file's overall lineage
+{"jsonrpc": "2.0", "id": 3, "method": "analyze_file", "params": {"file_path": "/path/to/file.py"}}
+```
+
+## Example
+
+```bash
+# Analyze the `authenticate` function in a FastAPI project
+GIT_REPO_PATH=/Users/fuads/fastapi arc analyze-function app/auth.py authenticate --repo fastapi/fastapi
+```
+
+Output:
+```
+Analyzing function authenticate in app/auth.py...
+Found 12 lineage edges for authenticate
+Summary: Found 12 historical versions of this code. Change types: physical: 8, identity: 4
+```
+
+With LLM synthesis:
+```
+The authenticate function evolved through 12 commits over 18 months. 
+Initial implementation used simple token validation, replaced in PR #2341 
+with OAuth2 Bearer token parsing after security audit. Several performance 
+optimizations were attempted (PRs #1892, #2103) but reverted due to race 
+conditions. The current implementation handles both JWT and opaque tokens 
+with a unified interface, consolidating three previous approaches.
+```
+
+## Testing
+
+```bash
+pytest tests/
+```
+
+## Roadmap
+
+- [x] Graph construction (GitWalker, ASTParser, LineageTracker)
+- [x] Contextual slicing (PRFetcher, GeographicFilter)
+- [x] CLI commands with local git repo auto-detection
+- [x] 10 language support (Python, JS, TS, Go, Rust, Java, C, C++, Ruby, PHP)
+- [x] Real-world testing on Flask repo
+- [x] MCP server (JSON-RPC 2.0 over stdio, works with Python 3.9)
+
+## License
+
+MIT

archeologist-0.1.0/archeologist.egg-info/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: archeologist
+Version: 0.1.0
+Summary: Code archaeologist - reconstruct function decision history via AST-aware lineage tracking
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: GitPython
+Requires-Dist: PyGithub
+Requires-Dist: tree-sitter>=0.21.0
+Requires-Dist: tree-sitter-languages>=1.9.0
+Requires-Dist: click
+Requires-Dist: pytest
+Requires-Dist: python-dotenv
+Requires-Dist: pyyaml
+
+# Archeologist - Semantic Lineage Graph Generator
+
+Post-incident archaeology tool that reconstructs function decision history via deterministic AST-aware lineage tracking.
+
+## CLI Usage
+
+```bash
+# Install
+pip install -e .
+
+# Analyze file (auto-detects git repo)
+arc analyze path/to/file.py
+
+# Analyze specific function  
+arc analyze-function path/to/file.py function_name
+
+# With GitHub PR integration
+arc analyze-function path/to/file.py function_name --repo owner/repo
+
+# With LLM narrative synthesis
+export CLAUDE_API_KEY=sk-ant-xxx
+arc analyze-function path/to/file.py function_name
+```
+
+## Configuration
+
+Set environment variables:
+
+```bash
+export GITHUB_TOKEN=ghp_xxx
+export CLAUDE_API_KEY=sk-ant-xxx
+export GIT_REPO_PATH=/path/to/local/repo
+```
+
+## Architecture
+
+Three-phase pipeline:
+
+1. **Semantic Lineage Tracking**
+   - GitWalker traverses history (--no-renames flag)
+   - ASTParser extracts function boundaries (Python, JS, TS, Go, Rust, Java, C, C++, Ruby, PHP)
+   - LineageTracker links nodes via four-tier hierarchy
+
+2. **Contextual Slicing**
+   - PRFetcher pulls associated PRs
+   - Geographic filter maps review comments to AST node line ranges
+
+3. **Narrative Synthesis**
+   - LiteLLM abstracts LLM calls (Claude, local models)
+   - Outputs 5-sentence brief explaining decisions
+
+## MCP Server
+
+The tool exposes an MCP-compatible JSON-RPC 2.0 server over stdio:
+
+```bash
+# Run MCP server
+arc-mcp
+
+# Or run directly
+python -m src.mcp.server
+```
+
+### Available Methods
+
+```json
+// List functions in a file
+{"jsonrpc": "2.0", "id": 1, "method": "list_functions", "params": {"file_path": "/path/to/file.py"}}
+
+// Analyze a specific function
+{"jsonrpc": "2.0", "id": 2, "method": "analyze_function", "params": {"file_path": "/path/to/file.py", "function_name": "foo"}}
+
+// Analyze a file's overall lineage
+{"jsonrpc": "2.0", "id": 3, "method": "analyze_file", "params": {"file_path": "/path/to/file.py"}}
+```
+
+## Example
+
+```bash
+# Analyze the `authenticate` function in a FastAPI project
+GIT_REPO_PATH=/Users/fuads/fastapi arc analyze-function app/auth.py authenticate --repo fastapi/fastapi
+```
+
+Output:
+```
+Analyzing function authenticate in app/auth.py...
+Found 12 lineage edges for authenticate
+Summary: Found 12 historical versions of this code. Change types: physical: 8, identity: 4
+```
+
+With LLM synthesis:
+```
+The authenticate function evolved through 12 commits over 18 months. 
+Initial implementation used simple token validation, replaced in PR #2341 
+with OAuth2 Bearer token parsing after security audit. Several performance 
+optimizations were attempted (PRs #1892, #2103) but reverted due to race 
+conditions. The current implementation handles both JWT and opaque tokens 
+with a unified interface, consolidating three previous approaches.
+```
+
+## Testing
+
+```bash
+pytest tests/
+```
+
+## Roadmap
+
+- [x] Graph construction (GitWalker, ASTParser, LineageTracker)
+- [x] Contextual slicing (PRFetcher, GeographicFilter)
+- [x] CLI commands with local git repo auto-detection
+- [x] 10 language support (Python, JS, TS, Go, Rust, Java, C, C++, Ruby, PHP)
+- [x] Real-world testing on Flask repo
+- [x] MCP server (JSON-RPC 2.0 over stdio, works with Python 3.9)
+
+## License
+
+MIT

archeologist-0.1.0/archeologist.egg-info/SOURCES.txt

@@ -0,0 +1,26 @@
+README.md
+pyproject.toml
+setup.cfg
+archeologist.egg-info/PKG-INFO
+archeologist.egg-info/SOURCES.txt
+archeologist.egg-info/dependency_links.txt
+archeologist.egg-info/entry_points.txt
+archeologist.egg-info/requires.txt
+archeologist.egg-info/top_level.txt
+src/__init__.py
+src/ast/__init__.py
+src/ast/lineage.py
+src/ast/parser.py
+src/cli/__init__.py
+src/cli/__main__.py
+src/db/__init__.py
+src/db/database.py
+src/git/__init__.py
+src/git/walker.py
+src/github/__init__.py
+src/github/fetcher.py
+src/github/geographic.py
+src/synthesis/__init__.py
+src/synthesis/narrative.py
+tests/test_e2e.py
+tests/test_lineage.py

archeologist-0.1.0/archeologist.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

archeologist-0.1.0/archeologist.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+arc = src.cli:cli

archeologist-0.1.0/archeologist.egg-info/requires.txt

@@ -0,0 +1,8 @@
+GitPython
+PyGithub
+tree-sitter>=0.21.0
+tree-sitter-languages>=1.9.0
+click
+pytest
+python-dotenv
+pyyaml

archeologist-0.1.0/archeologist.egg-info/top_level.txt

@@ -0,0 +1 @@
+src

archeologist-0.1.0/pyproject.toml

@@ -0,0 +1,31 @@
+[project]
+name = "archeologist"
+version = "0.1.0"
+description = "Code archaeologist - reconstruct function decision history via AST-aware lineage tracking"
+readme = "README.md"
+requires-python = ">=3.9"
+dependencies = [
+    "GitPython",
+    "PyGithub",
+    "tree-sitter>=0.21.0",
+    "tree-sitter-languages>=1.9.0",
+    "click",
+    "pytest",
+    "python-dotenv",
+    "pyyaml",
+]
+
+[project.scripts]
+arc = "src.cli:cli"
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["src*"]

archeologist-0.1.0/setup.cfg

@@ -0,0 +1,11 @@
+[tool:pytest]
+testpaths = tests
+python_files = test_*.py
+
+[coverage:run]
+source = src
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

archeologist-0.1.0/src/__init__.py

@@ -0,0 +1 @@
+# Archeologist - Code lineage tracking tool

archeologist-0.1.0/src/ast/__init__.py

@@ -0,0 +1 @@
+from .parser import ASTParser, ASTNode

archeologist-0.1.0/src/ast/lineage.py

@@ -0,0 +1,217 @@
+import re
+from dataclasses import dataclass
+from typing import Optional
+
+from ..git.walker import GitWalker, DiffHunk
+from ..ast.parser import ASTParser
+
+
+@dataclass
+class LineageEdge:
+    parent_node_id: str
+    child_node_id: str
+    change_type: str
+    confidence: float
+    commit_hash: str = ""
+    commit_message: str = ""
+    author: str = ""
+    date: str = ""
+
+
+class LineageTracker:
+    """Four-tier lineage detection for AST nodes across commits."""
+
+    def __init__(self, git_walker: GitWalker, ast_parser: ASTParser):
+        self.git = git_walker
+        self.ast = ast_parser
+
+    def track_lineage(
+        self, file_path: str, function_name: str, language: str, max_commits: int = 100
+    ) -> list[LineageEdge]:
+        """Track lineage of a function across git history."""
+        commits = self.git.get_commits_for_file(file_path, max_count=max_commits)
+        if not commits:
+            return []
+
+        commits = list(reversed(commits))
+
+        edges = []
+        previous_node = None
+        previous_content = None
+        previous_start_line = None
+
+        for commit in commits:
+            content = self.git.get_file_at_commit(commit.hash, file_path)
+            if not content:
+                continue
+
+            tree = self.ast.parse_file(content, language)
+            if not tree:
+                continue
+
+            current_node = self.ast.find_node_by_name(tree, language, function_name)
+            if not current_node:
+                continue
+
+            if previous_node and previous_content and previous_start_line is not None:
+                edge = self._detect_edge_type(
+                    previous_content,
+                    current_node.content,
+                    previous_start_line,
+                    current_node.start_line,
+                    previous_node.name,
+                    current_node.name,
+                    previous_node.node_id,
+                    current_node.node_id,
+                    commit.hash,
+                    file_path,
+                )
+                if edge:
+                    edge.commit_hash = commit.hash
+                    edge.commit_message = commit.message
+                    edge.author = commit.author
+                    edge.date = commit.date
+                    edges.append(edge)
+
+            previous_node = current_node
+            previous_content = current_node.content
+            previous_start_line = current_node.start_line
+
+        return edges
+
+    def _detect_edge_type(
+        self,
+        old_content: str,
+        new_content: str,
+        old_start_line: int,
+        new_start_line: int,
+        old_name: str,
+        new_name: str,
+        parent_id: str,
+        child_id: str,
+        commit_hash: str,
+        file_path: str,
+    ) -> Optional[LineageEdge]:
+        """Determine lineage edge type using four-tier hierarchy."""
+
+        tier1 = self._tier1_identity(old_content, new_content, parent_id, child_id)
+        if tier1:
+            return tier1
+
+        parent_commit = self.git.get_commit_parent(commit_hash)
+        if parent_commit:
+            diffs = self.git.get_diff_for_commit(commit_hash, file_path, parent_commit)
+            if diffs:
+                tier2 = self._tier2_physical(
+                    old_start_line, new_start_line, diffs, parent_id, child_id
+                )
+                if tier2:
+                    return tier2
+
+        tier3 = self._tier3_signature(
+            old_name, new_name, old_content, new_content, parent_id, child_id
+        )
+        if tier3:
+            return tier3
+
+        tier4 = self._tier4_semantic(old_content, new_content, parent_id, child_id)
+        if tier4:
+            return tier4
+
+        return None
+
+    def _tier1_identity(
+        self, old_content: str, new_content: str, parent_id: str, child_id: str
+    ) -> Optional[LineageEdge]:
+        """Tier 1: Global hash check for identical code."""
+        old_hash = self.git.compute_content_hash(old_content.strip())
+        new_hash = self.git.compute_content_hash(new_content.strip())
+
+        if old_hash == new_hash:
+            return LineageEdge(
+                parent_node_id=parent_id,
+                child_node_id=child_id,
+                change_type="identity",
+                confidence=1.0,
+            )
+        return None
+
+    def _tier2_physical(
+        self,
+        old_start_line: int,
+        new_start_line: int,
+        diffs: list[DiffHunk],
+        parent_id: str,
+        child_id: str,
+    ) -> Optional[LineageEdge]:
+        """Tier 2: Physical intersection via diff math."""
+        deleted = []
+        added = []
+        for diff in diffs:
+            deleted.extend(diff.deleted_lines)
+            added.extend(diff.added_lines)
+
+        line_shift = 0
+        for ln in added:
+            if ln < old_start_line:
+                line_shift += 1
+        for ln in deleted:
+            if ln < old_start_line:
+                line_shift -= 1
+
+        projected = old_start_line + line_shift
+        if abs(projected - new_start_line) <= 2:
+            return LineageEdge(
+                parent_node_id=parent_id,
+                child_node_id=child_id,
+                change_type="physical",
+                confidence=0.85,
+            )
+        return None
+
+    def _tier3_signature(
+        self,
+        old_name: str,
+        new_name: str,
+        old_content: str,
+        new_content: str,
+        parent_id: str,
+        child_id: str,
+    ) -> Optional[LineageEdge]:
+        """Tier 3: Signature match - name unchanged, content cut-and-paste."""
+        if old_name == new_name:
+            overlap = self._jaccard_similarity(
+                set(old_content.split()), set(new_content.split())
+            )
+            if 0.5 <= overlap:
+                return LineageEdge(
+                    parent_node_id=parent_id,
+                    child_node_id=child_id,
+                    change_type="signature",
+                    confidence=overlap,
+                )
+        return None
+
+    def _tier4_semantic(
+        self, old_content: str, new_content: str, parent_id: str, child_id: str
+    ) -> Optional[LineageEdge]:
+        """Tier 4: Semantic overlap - Jaccard for split/merge."""
+        overlap = self._jaccard_similarity(
+            set(old_content.split()), set(new_content.split())
+        )
+        if overlap >= 0.5:
+            return LineageEdge(
+                parent_node_id=parent_id,
+                child_node_id=child_id,
+                change_type="semantic",
+                confidence=overlap,
+            )
+        return None
+
+    def _jaccard_similarity(self, set1: set, set2: set) -> float:
+        """Calculate Jaccard similarity between two sets."""
+        if not set1 or not set2:
+            return 0.0
+        intersection = len(set1 & set2)
+        union = len(set1 | set2)
+        return intersection / union if union > 0 else 0.0