aidp 0.3.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ceebdfcb742085ac8d9ea6b5db63557d5fe999f498e97699c199fc9651cf4654
-  data.tar.gz: f1b230dcc33692c4c0391464beaf305b76993dbe02964d8c80198a7a50294a5c
+  metadata.gz: e32a892070257857ac578c965a7aa1bb240a1a40a58f884bc6740a5aa6a57804
+  data.tar.gz: cfca9c65bc0d3b5e784fbab8dcff12545982d5bb00057e575dfdfcd77007accf
 SHA512:
-  metadata.gz: d20547fa251feb09a97e6795b04fc09dd13d0f0fd801c5abdc994bb27a20898a604692eb04b7ed8a370ac7619bcbaf89a17af60dc34c844e782e8b68087ef67a
-  data.tar.gz: cb8503f9b0a85d6113d9ab2dada1cb0087b9b9e6ab1abb44894aa5152d3596c782c08f4708e3f03f9b969a94736acb7d882350b6f0cb869787069c8b12ddde85
+  metadata.gz: 99c4b48fce7af4bc8a043549fcb4ba0119dc84af870fdb8b0aae26045c4626e16cbd035df94e8b5d5c638d10a1a26f3a76b7c260647b701a55ef1f795a373ca0
+  data.tar.gz: 818054559441866f659d77d5b0e10721822c3aa5abc74010e90f5cbbb13a3d1fe85355fb6e2f82a867e24ed4b24e71670a72370b23c69526bf0400f7a1bc2a6a

data/README.md

@@ -76,6 +76,7 @@ The pipeline includes 15 steps total:
 aidp status                   # Show progress of all steps
 aidp execute next             # Run next pending step
 aidp approve current          # Approve current gate step
+aidp jobs                     # Monitor background jobs (real-time)
 aidp detect                   # See which AI provider will be used
 aidp execute <step>           # Run specific step (e.g., prd, arch, tasks)
 aidp approve <step>           # Approve specific step
@@ -97,6 +98,164 @@ AIDP_PROVIDER=anthropic aidp execute next
 AIDP_LLM_CMD=/usr/local/bin/claude aidp execute next
 ```
 
+## Tree-sitter Static Analysis
+
+AIDP includes powerful Tree-sitter-based static analysis capabilities for code.
+
+### Tree-sitter Dependencies
+
+The Tree-sitter analysis requires the Tree-sitter system library and pre-compiled language parsers:
+
+```bash
+# Install Tree-sitter system library
+# macOS
+brew install tree-sitter
+
+# Ubuntu/Debian
+sudo apt-get install tree-sitter
+
+# Or follow the ruby_tree_sitter README for other platforms
+# https://github.com/Faveod/ruby-tree-sitter#installation
+
+# Install Tree-sitter parsers
+./install_tree_sitter_parsers.sh
+```
+
+### Parser Installation Script
+
+The `install_tree_sitter_parsers.sh` script automatically downloads and installs pre-built Tree-sitter parsers:
+
+```bash
+# Make the script executable
+chmod +x install_tree_sitter_parsers.sh
+
+# Run the installation script
+./install_tree_sitter_parsers.sh
+```
+
+The script will:
+
+- Detect your OS and architecture (macOS ARM64, Linux x64, etc.)
+- Download the appropriate parser bundle from [Faveod/tree-sitter-parsers](https://github.com/Faveod/tree-sitter-parsers/releases/tag/v4.9)
+- Extract parsers to `.aidp/parsers/` directory
+- Set up the `TREE_SITTER_PARSERS` environment variable
+
+### Environment Setup
+
+After running the installation script, make the environment variable permanent:
+
+```bash
+# Add to your shell profile (e.g., ~/.zshrc, ~/.bashrc)
+echo 'export TREE_SITTER_PARSERS="$(pwd)/.aidp/parsers"' >> ~/.zshrc
+
+# Reload your shell
+source ~/.zshrc
+```
+
+### Tree-sitter Analysis Commands
+
+```bash
+# Run Tree-sitter static analysis
+aidp analyze code
+
+# Analyze specific languages
+aidp analyze code --langs ruby,javascript,typescript
+
+# Use multiple threads for faster analysis
+aidp analyze code --threads 8
+
+# Rebuild knowledge base from scratch
+aidp analyze code --rebuild
+
+# Specify custom KB directory
+aidp analyze code --kb-dir .aidp/custom-kb
+
+# Inspect generated knowledge base
+aidp kb show
+
+# Show specific KB data
+aidp kb show symbols
+aidp kb show imports
+aidp kb show seams
+
+# Generate dependency graphs
+aidp kb graph imports
+aidp kb graph calls
+```
+
+### Knowledge Base Structure
+
+The Tree-sitter analysis generates structured JSON files in `.aidp/kb/`:
+
+- **`symbols.json`** - Classes, modules, methods, and their metadata
+- **`imports.json`** - Require statements and dependencies
+- **`calls.json`** - Method calls and invocation patterns
+- **`metrics.json`** - Code complexity and size metrics
+- **`seams.json`** - Integration points and dependency injection opportunities
+- **`hotspots.json`** - Frequently changed code areas (based on git history)
+- **`tests.json`** - Test coverage analysis
+- **`cycles.json`** - Circular dependency detection
+
+### Legacy Code Analysis Features
+
+The Tree-sitter analysis specifically supports:
+
+- **Seam Detection**: Identifies I/O operations, global state access, and constructor dependencies
+- **Change Hotspots**: Uses git history to identify frequently modified code
+- **Dependency Analysis**: Maps import relationships and call graphs
+- **Test Coverage**: Identifies untested public APIs
+- **Refactoring Opportunities**: Suggests dependency injection points and seam locations
+
+## Background Jobs
+
+AIDP uses background jobs to handle all AI provider executions, providing better reliability and real-time monitoring capabilities.
+
+### Job Monitoring
+
+Monitor running and completed jobs in real-time:
+
+```bash
+aidp jobs                     # Show job status with real-time updates
+```
+
+The jobs view displays:
+
+- **Running jobs** with live progress updates
+- **Queued jobs** waiting to be processed
+- **Completed jobs** with execution results
+- **Failed jobs** with error details
+
+### Job Controls
+
+From the jobs view, you can:
+
+- **Retry failed jobs** by pressing `r` on a failed job
+- **View job details** by pressing `d` on any job
+- **Exit monitoring** by pressing `q`
+
+### Job Persistence
+
+- Jobs persist across CLI restarts
+- Job history is preserved for analysis
+- Failed jobs can be retried at any time
+- All job metadata and logs are stored
+
+### Database Setup
+
+AIDP uses PostgreSQL for job management. Ensure PostgreSQL is installed and running:
+
+```bash
+# macOS (using Homebrew)
+brew install postgresql
+brew services start postgresql
+
+# Ubuntu/Debian
+sudo apt-get install postgresql postgresql-contrib
+sudo systemctl start postgresql
+
+# The database will be created automatically on first use
+```
+
 ## File-Based Interaction
 
 At gate steps, the AI creates files for interaction instead of requiring real-time chat:
@@ -137,20 +296,24 @@ Here's a typical session:
 aidp execute next
 # → Creates docs/PRD.md and PRD_QUESTIONS.md
 
-# 2. Review the questions (if any)
+# 2. Monitor job progress (optional)
+aidp jobs
+# → Shows real-time job status and progress
+
+# 3. Review the questions (if any)
 cat PRD_QUESTIONS.md
 # → If questions exist, edit the file with your answers, then re-run
 
-# 3. Review the PRD
+# 4. Review the PRD
 cat docs/PRD.md
 # → Edit if needed
 
-# 4. Approve and continue
+# 5. Approve and continue
 aidp approve current
 aidp execute next
 # → Creates docs/NFRs.md automatically
 
-# 5. Continue through gates
+# 6. Continue through gates
 aidp execute next
 # → Creates docs/Architecture.md and ARCH_QUESTIONS.md
 # → Repeat review/approve cycle
@@ -162,9 +325,19 @@ aidp execute next
 # Install dependencies
 bundle install
 
+# Install Tree-sitter parsers for development
+./install_tree_sitter_parsers.sh
+
+# Set up environment variables
+export TREE_SITTER_PARSERS="$(pwd)/.aidp/parsers"
+
 # Run tests
 bundle exec rspec
 
+# Run Tree-sitter analysis tests specifically
+bundle exec rspec spec/aidp/analysis/
+bundle exec rspec spec/integration/tree_sitter_analysis_workflow_spec.rb
+
 # Run linter
 bundle exec standardrb
 
@@ -175,6 +348,19 @@ bundle exec standardrb --fix
 bundle exec rake build
 ```
 
+### Development Dependencies
+
+The following system dependencies are required for development:
+
+- **Tree-sitter** - System library for parsing (install via `brew install tree-sitter` or package manager)
+- **PostgreSQL** - Database for job management
+- **Ruby gems** - All required gems are specified in `aidp.gemspec` and installed via `bundle install`
+
+Optional gems with fallbacks:
+
+- **`concurrent-ruby`** - Parallel processing (fallback to basic threading if not available)
+- **`tty-table`** - Table rendering (fallback to basic ASCII tables if not available)
+
 ## Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and conventional commit guidelines.
@@ -199,7 +385,7 @@ The gem automates a complete 15-step development pipeline:
 - **Threat Model** → Security analysis (`docs/ThreatModel.md`)
 - **Test Plan** → Testing strategy (`docs/TestPlan.md`)
 - **Scaffolding** → Project structure guidance (`docs/ScaffoldingGuide.md`)
-- **Static Analysis** → Code quality tools (`docs/StaticAnalysis.md`)
+- **Static Analysis** → Code quality tools and Tree-sitter analysis (`docs/StaticAnalysis.md`, `.aidp/kb/`)
 - **Observability** → Monitoring and SLOs (`docs/Observability.md`)
 - **Delivery** → Deployment strategy (`docs/DeliveryPlan.md`)
 - **Docs Portal** → Documentation portal (`docs/DocsPortalPlan.md`)

data/lib/aidp/analysis/kb_inspector.rb

@@ -0,0 +1,456 @@
+# frozen_string_literal: true
+
+require "json"
+require "tty-table"
+
+module Aidp
+  module Analysis
+    class KBInspector
+      def initialize(kb_dir = ".aidp/kb")
+        @kb_dir = File.expand_path(kb_dir)
+        @data = load_kb_data
+      end
+
+      def show(type, format: "summary")
+        case type
+        when "seams"
+          show_seams(format)
+        when "hotspots"
+          show_hotspots(format)
+        when "cycles"
+          show_cycles(format)
+        when "apis"
+          show_apis(format)
+        when "symbols"
+          show_symbols(format)
+        when "imports"
+          show_imports(format)
+        when "summary"
+          show_summary(format)
+        else
+          puts "Unknown KB type: #{type}"
+          puts "Available types: seams, hotspots, cycles, apis, symbols, imports, summary"
+        end
+      end
+
+      def generate_graph(type, format: "dot", output: nil)
+        case type
+        when "imports"
+          generate_import_graph(format, output)
+        when "calls"
+          generate_call_graph(format, output)
+        when "cycles"
+          generate_cycle_graph(format, output)
+        else
+          puts "Unknown graph type: #{type}"
+          puts "Available types: imports, calls, cycles"
+        end
+      end
+
+      private
+
+      def truncate_text(text, max_length = 50)
+        return nil unless text
+        return text if text.length <= max_length
+
+        text[0..max_length - 4] + "..."
+      end
+
+      def create_table(header, rows)
+        TTY::Table.new(header: header, rows: rows)
+      end
+
+      def load_kb_data
+        data = {}
+
+        %w[symbols imports calls metrics seams hotspots tests cycles].each do |type|
+          file_path = File.join(@kb_dir, "#{type}.json")
+          if File.exist?(file_path)
+            begin
+              data[type.to_sym] = JSON.parse(File.read(file_path), symbolize_names: true)
+            rescue JSON::ParserError => e
+              puts "Warning: Could not parse #{file_path}: #{e.message}"
+              data[type.to_sym] = []
+            end
+          else
+            data[type.to_sym] = []
+          end
+        end
+
+        data
+      end
+
+      def show_summary(_format)
+        puts "\n📊 Knowledge Base Summary"
+        puts "=" * 50
+
+        puts "📁 KB Directory: #{@kb_dir}"
+        puts "📄 Files analyzed: #{count_files}"
+        puts "🏗️  Symbols: #{@data[:symbols]&.length || 0}"
+        puts "📦 Imports: #{@data[:imports]&.length || 0}"
+        puts "🔗 Calls: #{@data[:calls]&.length || 0}"
+        puts "📏 Metrics: #{@data[:metrics]&.length || 0}"
+        puts "🔧 Seams: #{@data[:seams]&.length || 0}"
+        puts "🔥 Hotspots: #{@data[:hotspots]&.length || 0}"
+        puts "🧪 Tests: #{@data[:tests]&.length || 0}"
+        puts "🔄 Cycles: #{@data[:cycles]&.length || 0}"
+
+        if @data[:seams]&.any?
+          puts "\n🔧 Seam Types:"
+          seam_types = @data[:seams].group_by { |s| s[:kind] }
+          seam_types.each do |type, seams|
+            puts "  #{type}: #{seams.length}"
+          end
+        end
+
+        if @data[:hotspots]&.any?
+          puts "\n🔥 Top 5 Hotspots:"
+          @data[:hotspots].first(5).each_with_index do |hotspot, i|
+            puts "  #{i + 1}. #{hotspot[:file]}:#{hotspot[:method]} (score: #{hotspot[:score]})"
+          end
+        end
+      end
+
+      def show_seams(format)
+        return puts "No seams data available" unless @data[:seams]&.any?
+
+        case format
+        when "json"
+          puts JSON.pretty_generate(@data[:seams])
+        when "table"
+          show_seams_table
+        else
+          show_seams_summary
+        end
+      end
+
+      def show_seams_table
+        table = create_table(
+          ["Type", "File", "Line", "Symbol", "Suggestion"],
+          @data[:seams].map do |seam|
+            [
+              seam[:kind],
+              seam[:file],
+              seam[:line],
+              seam[:symbol_id]&.split(":")&.last || "N/A",
+              truncate_text(seam[:suggestion], 50) || "N/A"
+            ]
+          end
+        )
+
+        puts "\n🔧 Seams Analysis"
+        puts "=" * 80
+        puts table.render
+      end
+
+      def show_seams_summary
+        puts "\n🔧 Seams Analysis"
+        puts "=" * 50
+
+        seam_types = @data[:seams].group_by { |s| s[:kind] }
+
+        seam_types.each do |type, seams|
+          puts "\n📌 #{type.upcase} (#{seams.length} found)"
+          puts "-" * 30
+
+          seams.first(10).each do |seam|
+            puts "  #{seam[:file]}:#{seam[:line]}"
+            puts "    Symbol: #{seam[:symbol_id]&.split(":")&.last}"
+            puts "    Suggestion: #{seam[:suggestion]}"
+            puts
+          end
+
+          if seams.length > 10
+            puts "  ... and #{seams.length - 10} more"
+          end
+        end
+      end
+
+      def show_hotspots(format)
+        return puts "No hotspots data available" unless @data[:hotspots]&.any?
+
+        case format
+        when "json"
+          puts JSON.pretty_generate(@data[:hotspots])
+        when "table"
+          show_hotspots_table
+        else
+          show_hotspots_summary
+        end
+      end
+
+      def show_hotspots_table
+        table = create_table(
+          ["Rank", "File", "Method", "Score", "Complexity", "Touches"],
+          @data[:hotspots].map.with_index do |hotspot, i|
+            [
+              i + 1,
+              hotspot[:file],
+              hotspot[:method],
+              hotspot[:score],
+              hotspot[:complexity],
+              hotspot[:touches]
+            ]
+          end
+        )
+
+        puts "\n🔥 Code Hotspots"
+        puts "=" * 80
+        puts table.render
+      end
+
+      def show_hotspots_summary
+        puts "\n🔥 Code Hotspots (Top 20)"
+        puts "=" * 50
+
+        @data[:hotspots].each_with_index do |hotspot, i|
+          puts "#{i + 1}. #{hotspot[:file]}:#{hotspot[:method]}"
+          puts "   Score: #{hotspot[:score]} (Complexity: #{hotspot[:complexity]}, Touches: #{hotspot[:touches]})"
+          puts
+        end
+      end
+
+      def show_cycles(format)
+        return puts "No cycles data available" unless @data[:cycles]&.any?
+
+        case format
+        when "json"
+          puts JSON.pretty_generate(@data[:cycles])
+        else
+          show_cycles_summary
+        end
+      end
+
+      def show_cycles_summary
+        puts "\n🔄 Import Cycles"
+        puts "=" * 50
+
+        @data[:cycles].each_with_index do |cycle, i|
+          puts "Cycle #{i + 1}:"
+          cycle[:members].each do |member|
+            puts "  - #{member}"
+          end
+          puts "  Weight: #{cycle[:weight]}" if cycle[:weight]
+          puts
+        end
+      end
+
+      def show_apis(format)
+        return puts "No APIs data available" unless @data[:tests]&.any?
+
+        untested_apis = @data[:tests].select { |t| t[:tests].empty? }
+
+        case format
+        when "json"
+          puts JSON.pretty_generate(untested_apis)
+        else
+          show_apis_summary(untested_apis)
+        end
+      end
+
+      def show_apis_summary(untested_apis)
+        puts "\n🧪 Untested Public APIs"
+        puts "=" * 50
+
+        if untested_apis.empty?
+          puts "✅ All public APIs have associated tests!"
+        else
+          puts "Found #{untested_apis.length} untested public APIs:"
+          puts
+
+          untested_apis.each do |api|
+            symbol = @data[:symbols]&.find { |s| s[:id] == api[:symbol_id] }
+            if symbol
+              puts "  #{symbol[:file]}:#{symbol[:line]} - #{symbol[:name]}"
+              puts "    Suggestion: Create characterization tests"
+              puts
+            end
+          end
+        end
+      end
+
+      def show_symbols(format)
+        return puts "No symbols data available" unless @data[:symbols]&.any?
+
+        case format
+        when "json"
+          puts JSON.pretty_generate(@data[:symbols])
+        when "table"
+          show_symbols_table
+        else
+          show_symbols_summary
+        end
+      end
+
+      def show_symbols_table
+        table = create_table(
+          ["Type", "Name", "File", "Line", "Visibility"],
+          @data[:symbols].map do |symbol|
+            [
+              symbol[:kind],
+              symbol[:name],
+              symbol[:file],
+              symbol[:line],
+              symbol[:visibility]
+            ]
+          end
+        )
+
+        puts "\n🏗️  Symbols"
+        puts "=" * 80
+        puts table.render
+      end
+
+      def show_symbols_summary
+        puts "\n🏗️  Symbols Summary"
+        puts "=" * 50
+
+        symbol_types = @data[:symbols].group_by { |s| s[:kind] }
+
+        symbol_types.each do |type, symbols|
+          puts "#{type.capitalize}: #{symbols.length}"
+        end
+      end
+
+      def show_imports(format)
+        return puts "No imports data available" unless @data[:imports]&.any?
+
+        case format
+        when "json"
+          puts JSON.pretty_generate(@data[:imports])
+        when "table"
+          show_imports_table
+        else
+          show_imports_summary
+        end
+      end
+
+      def show_imports_table
+        table = create_table(
+          ["Type", "Target", "File", "Line"],
+          @data[:imports].map do |import|
+            [
+              import[:kind],
+              import[:target],
+              import[:file],
+              import[:line]
+            ]
+          end
+        )
+
+        puts "\n📦 Imports"
+        puts "=" * 80
+        puts table.render
+      end
+
+      def show_imports_summary
+        puts "\n📦 Imports Summary"
+        puts "=" * 50
+
+        import_types = @data[:imports].group_by { |i| i[:kind] }
+
+        import_types.each do |type, imports|
+          puts "#{type.capitalize}: #{imports.length}"
+        end
+      end
+
+      def generate_import_graph(format, output)
+        puts "Generating import graph in #{format} format..."
+
+        case format
+        when "dot"
+          generate_dot_graph(output)
+        when "mermaid"
+          generate_mermaid_graph(output)
+        when "json"
+          generate_json_graph(output)
+        else
+          puts "Unsupported graph format: #{format}"
+        end
+      end
+
+      def generate_dot_graph(output)
+        content = ["digraph ImportGraph {"]
+        content << "  rankdir=LR;"
+        content << "  node [shape=box];"
+
+        @data[:imports]&.each do |import|
+          from = import[:file].gsub(/[^a-zA-Z0-9]/, "_")
+          to = import[:target].gsub(/[^a-zA-Z0-9]/, "_")
+          content << "  \"#{from}\" -> \"#{to}\" [label=\"#{import[:kind]}\"];"
+        end
+
+        content << "}"
+
+        if output
+          File.write(output, content.join("\n"))
+          puts "Graph written to #{output}"
+        else
+          puts content.join("\n")
+        end
+      end
+
+      def generate_mermaid_graph(output)
+        content = ["graph LR"]
+
+        @data[:imports]&.each do |import|
+          from = import[:file].gsub(/[^a-zA-Z0-9]/, "_")
+          to = import[:target].gsub(/[^a-zA-Z0-9]/, "_")
+          content << "  #{from} --> #{to}"
+        end
+
+        if output
+          File.write(output, content.join("\n"))
+          puts "Graph written to #{output}"
+        else
+          puts content.join("\n")
+        end
+      end
+
+      def generate_json_graph(output)
+        graph_data = {
+          nodes: [],
+          edges: []
+        }
+
+        # Add nodes
+        files = (@data[:imports]&.map { |i| i[:file] } || []).uniq
+        targets = (@data[:imports]&.map { |i| i[:target] } || []).uniq
+
+        (files + targets).uniq.each do |node|
+          graph_data[:nodes] << {id: node, label: node}
+        end
+
+        # Add edges
+        @data[:imports]&.each do |import|
+          graph_data[:edges] << {
+            from: import[:file],
+            to: import[:target],
+            label: import[:kind]
+          }
+        end
+
+        if output
+          File.write(output, JSON.pretty_generate(graph_data))
+          puts "Graph written to #{output}"
+        else
+          puts JSON.pretty_generate(graph_data)
+        end
+      end
+
+      def generate_call_graph(_format, _output)
+        # Similar to import graph but for method calls
+        puts "Call graph generation not yet implemented"
+      end
+
+      def generate_cycle_graph(_format, _output)
+        # Generate graph showing only the cycles
+        puts "Cycle graph generation not yet implemented"
+      end
+
+      def count_files
+        @data[:symbols]&.map { |s| s[:file] }&.uniq&.length || 0
+      end
+    end
+  end
+end