@voodocs/cli 2.4.0 → 2.5.1

package/CHANGELOG.md

@@ -1,5 +1,52 @@
 # CHANGELOG
 
+## [2.5.1] - 2024-12-24
+
+### Fixed
+- **Packaging Issue**: Fixed npm package to include missing files from v2.5.0
+  - Added `lib/darkarts/priority_analyzer/` directory
+  - Added `lib/darkarts/voodocs_lite_dict.py`
+  - Added `lib/darkarts/voodocs_lite_dict_v2.py`
+  - Added `lib/darkarts/voodocs_lite_parser.py`
+- v2.5.0 was published with CHANGELOG updates but missing actual code files
+- All Priority Analyzer and VooDocs Lite features now properly included
+
+---
+
+## [2.5.0] - 2024-12-24
+
+### Added
+- **Priority Analyzer** - New `voodocs analyze` command to identify which files need annotations most urgently
+  - **Complexity Analysis**: LOC, cyclomatic complexity, function count
+  - **Security Keyword Detection**: auth, password, admin, crypto, SQL, payment, etc.
+  - **Dependency Analysis**: Import tracking and dependent file detection
+  - **Weighted Scoring**: Complexity (30%), Security (40%), Dependencies (20%), Annotations (10%)
+  - **Priority Levels**: CRITICAL (80-100), HIGH (60-79), MEDIUM (40-59), LOW (20-39), MINIMAL (0-19)
+  - **Multiple Output Formats**: text, JSON, CSV, Markdown
+  - **Filtering Options**: By priority level, minimum score, exclude patterns
+  - **Smart Suggestions**: Context-aware recommendations for each file
+- **Priority Analyzer Modules**:
+  - `lib/darkarts/priority_analyzer/complexity.py` - Complexity analysis engine
+  - `lib/darkarts/priority_analyzer/security.py` - Security keyword detector
+  - `lib/darkarts/priority_analyzer/dependencies.py` - Dependency tracker
+  - `lib/darkarts/priority_analyzer/analyzer.py` - Main priority analyzer
+  - `lib/cli/analyze.py` - CLI command implementation
+- **Comprehensive Test Suite** - 16 unit tests for priority analyzer functionality
+
+### Improved
+- Heuristics for identifying security-sensitive code
+- Dependency tracking for TypeScript, JavaScript, Python, and Solidity
+- Annotation coverage detection and penalty calculation
+- Priority scoring algorithm with empirical validation
+
+### Documentation
+- Added Priority Analyzer section to README.md
+- Added analyze command documentation with examples
+- Added priority level definitions and scoring factors
+- Added specification document for priority analyzer
+
+---
+
 ## [2.4.0] - 2025-12-24
 
 ### Added

package/README.md

@@ -348,6 +348,32 @@ voodocs convert src/ --to lite --dry-run  # Preview changes
 voodocs convert src/ --to lite --in-place # Modify files in-place
 ```
 
+### `voodocs analyze`
+
+**NEW in v2.5.0!** Analyze files and suggest annotation priorities based on complexity, security, and dependencies:
+
+```bash
+voodocs analyze src/                # Analyze directory
+voodocs analyze src/ --top 20       # Show top 20 files
+voodocs analyze src/ --priority critical  # Filter by priority
+voodocs analyze src/ --format json  # JSON output
+voodocs analyze src/ --min-score 60 # Only show files with score >= 60
+voodocs analyze src/ --exclude node_modules,dist  # Exclude patterns
+```
+
+**Priority Levels:**
+- 🔴 **CRITICAL** (80-100): Highly complex, security-sensitive, or widely-used files
+- 🟠 **HIGH** (60-79): Complex or security-sensitive files
+- 🟡 **MEDIUM** (40-59): Moderately complex files
+- 🟢 **LOW** (20-39): Simple files with some complexity
+- ⚪ **MINIMAL** (0-19): Very simple files
+
+**Scoring Factors:**
+- **Complexity** (30%): Lines of code, cyclomatic complexity, function count
+- **Security** (40%): Security-sensitive keywords (auth, password, admin, etc.)
+- **Dependencies** (20%): Import count and dependent files
+- **Annotations** (10%): Penalty for missing VooDocs annotations
+
 ### `voodocs generate`
 
 Generate documentation from annotations:

package/lib/cli/analyze.py

@@ -0,0 +1,277 @@
+"""
+CLI command for VooDocs Priority Analyzer
+
+Usage: voodocs analyze [path] [options]
+"""
+
+import os
+import sys
+from pathlib import Path
+
+# Add parent directory to path for imports
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../..')))
+
+from lib.darkarts.priority_analyzer.analyzer import PriorityAnalyzer
+
+
+def cmd_analyze(args):
+    """Execute the analyze command."""
+    # Get path (default to current directory)
+    path = args.path if hasattr(args, 'path') and args.path else '.'
+    path = os.path.abspath(path)
+    
+    # Check if path exists
+    if not os.path.exists(path):
+        print(f"❌ Error: Path not found: {path}")
+        return 1
+    
+    # Determine if path is file or directory
+    is_file = os.path.isfile(path)
+    
+    # Initialize analyzer
+    project_root = path if not is_file else str(Path(path).parent)
+    analyzer = PriorityAnalyzer(project_root=project_root)
+    
+    # Analyze
+    if is_file:
+        scores = [analyzer.analyze_file(path)]
+    else:
+        recursive = getattr(args, 'recursive', True)
+        exclude = getattr(args, 'exclude', None)
+        exclude_patterns = exclude.split(',') if exclude else None
+        
+        print(f"🔍 Analyzing files in: {path}")
+        if recursive:
+            print("   (recursive mode)")
+        print()
+        
+        scores = analyzer.analyze_directory(path, recursive, exclude_patterns)
+    
+    # Filter by priority level if specified
+    if hasattr(args, 'priority') and args.priority:
+        priority_filter = args.priority.upper()
+        scores = [s for s in scores if s.priority_level == priority_filter]
+    
+    # Filter by minimum score
+    if hasattr(args, 'min_score') and args.min_score:
+        scores = [s for s in scores if s.priority_score >= args.min_score]
+    
+    # Filter out fully annotated files (unless --include-annotated)
+    if not getattr(args, 'include_annotated', False):
+        scores = [s for s in scores if s.annotation_coverage < 1.0]
+    
+    # Limit to top N
+    top_n = getattr(args, 'top', 10)
+    if top_n and top_n > 0:
+        scores = scores[:top_n]
+    
+    # Output format
+    output_format = getattr(args, 'format', 'text')
+    
+    if output_format == 'json':
+        _output_json(scores)
+    elif output_format == 'csv':
+        _output_csv(scores)
+    elif output_format == 'markdown':
+        _output_markdown(scores)
+    else:
+        _output_text(scores, path)
+    
+    return 0
+
+
+def _output_text(scores, path):
+    """Output results in human-readable text format."""
+    if not scores:
+        print("✅ No files need annotations!")
+        print("   All files are either fully annotated or below priority threshold.")
+        return
+    
+    # Calculate statistics
+    total_files = len(scores)
+    priority_dist = {
+        'CRITICAL': len([s for s in scores if s.priority_level == 'CRITICAL']),
+        'HIGH': len([s for s in scores if s.priority_level == 'HIGH']),
+        'MEDIUM': len([s for s in scores if s.priority_level == 'MEDIUM']),
+        'LOW': len([s for s in scores if s.priority_level == 'LOW']),
+        'MINIMAL': len([s for s in scores if s.priority_level == 'MINIMAL']),
+    }
+    
+    annotated_count = sum(1 for s in scores if s.annotation_coverage > 0)
+    coverage_pct = (annotated_count / total_files * 100) if total_files > 0 else 0
+    
+    # Header
+    print("━" * 70)
+    print("VooDocs Priority Analysis")
+    print("━" * 70)
+    print()
+    print(f"Project: {path}")
+    print(f"Files analyzed: {total_files}")
+    print(f"Annotations found: {annotated_count} ({coverage_pct:.0f}% coverage)")
+    print()
+    
+    # Priority distribution
+    print("Priority Distribution:")
+    if priority_dist['CRITICAL'] > 0:
+        print(f"🔴 CRITICAL (80-100): {priority_dist['CRITICAL']} files")
+    if priority_dist['HIGH'] > 0:
+        print(f"🟠 HIGH (60-79):     {priority_dist['HIGH']} files")
+    if priority_dist['MEDIUM'] > 0:
+        print(f"🟡 MEDIUM (40-59):   {priority_dist['MEDIUM']} files")
+    if priority_dist['LOW'] > 0:
+        print(f"🟢 LOW (20-39):      {priority_dist['LOW']} files")
+    if priority_dist['MINIMAL'] > 0:
+        print(f"⚪ MINIMAL (0-19):    {priority_dist['MINIMAL']} files")
+    print()
+    
+    # Top files
+    print("━" * 70)
+    print(f"Top {len(scores)} Files Needing Annotations")
+    print("━" * 70)
+    print()
+    
+    for i, score in enumerate(scores, 1):
+        # Priority emoji
+        emoji = {
+            'CRITICAL': '🔴',
+            'HIGH': '🟠',
+            'MEDIUM': '🟡',
+            'LOW': '🟢',
+            'MINIMAL': '⚪'
+        }.get(score.priority_level, '⚪')
+        
+        # Relative path
+        rel_path = os.path.relpath(score.filepath, path) if not os.path.isfile(path) else score.filepath
+        
+        print(f"{i}. {emoji} {rel_path} (Score: {score.priority_score:.0f})")
+        print(f"   Complexity: {score.complexity_score}  Security: {score.security_score}  Dependencies: {score.dependency_score}  Annotations: {score.annotation_penalty}")
+        print()
+        
+        # Reasons
+        if score.reasons:
+            print("   Reasons:")
+            for reason in score.reasons:
+                print(f"   • {reason}")
+            print()
+        
+        # Suggestions
+        if score.suggestions:
+            print("   Suggestions:")
+            for suggestion in score.suggestions:
+                print(f"   ✓ {suggestion}")
+            print()
+    
+    # Footer
+    print("━" * 70)
+    print()
+    print("💡 Tip: Start with CRITICAL files first. Use:")
+    if scores:
+        first_file = scores[0].filepath
+        print(f"   voodocs companion {first_file}")
+    print()
+    print("   Or convert to VooDocs Lite for faster annotation:")
+    print(f"   voodocs convert {path} --to lite --in-place")
+    print()
+
+
+def _output_json(scores):
+    """Output results in JSON format."""
+    import json
+    
+    data = []
+    for score in scores:
+        data.append({
+            'filepath': score.filepath,
+            'priority_score': score.priority_score,
+            'priority_level': score.priority_level,
+            'complexity_score': score.complexity_score,
+            'security_score': score.security_score,
+            'dependency_score': score.dependency_score,
+            'annotation_penalty': score.annotation_penalty,
+            'annotation_coverage': score.annotation_coverage,
+            'loc': score.loc,
+            'functions': score.functions,
+            'security_keywords': score.security_keywords,
+            'dependent_count': score.dependent_count,
+            'reasons': score.reasons,
+            'suggestions': score.suggestions
+        })
+    
+    print(json.dumps(data, indent=2))
+
+
+def _output_csv(scores):
+    """Output results in CSV format."""
+    import csv
+    import sys
+    
+    writer = csv.writer(sys.stdout)
+    
+    # Header
+    writer.writerow([
+        'filepath', 'priority_score', 'priority_level',
+        'complexity_score', 'security_score', 'dependency_score',
+        'annotation_penalty', 'annotation_coverage',
+        'loc', 'functions', 'dependent_count'
+    ])
+    
+    # Data
+    for score in scores:
+        writer.writerow([
+            score.filepath,
+            score.priority_score,
+            score.priority_level,
+            score.complexity_score,
+            score.security_score,
+            score.dependency_score,
+            score.annotation_penalty,
+            score.annotation_coverage,
+            score.loc,
+            score.functions,
+            score.dependent_count
+        ])
+
+
+def _output_markdown(scores):
+    """Output results in Markdown format."""
+    print("# VooDocs Priority Analysis")
+    print()
+    print("## Summary")
+    print()
+    print(f"- **Files analyzed:** {len(scores)}")
+    print()
+    print("## Top Files")
+    print()
+    
+    for i, score in enumerate(scores, 1):
+        emoji = {
+            'CRITICAL': '🔴',
+            'HIGH': '🟠',
+            'MEDIUM': '🟡',
+            'LOW': '🟢',
+            'MINIMAL': '⚪'
+        }.get(score.priority_level, '⚪')
+        
+        print(f"### {i}. {emoji} {score.filepath}")
+        print()
+        print(f"**Score:** {score.priority_score:.0f} ({score.priority_level})")
+        print()
+        print(f"- Complexity: {score.complexity_score}")
+        print(f"- Security: {score.security_score}")
+        print(f"- Dependencies: {score.dependency_score}")
+        print(f"- Annotation Coverage: {score.annotation_coverage * 100:.0f}%")
+        print()
+        
+        if score.reasons:
+            print("**Reasons:**")
+            print()
+            for reason in score.reasons:
+                print(f"- {reason}")
+            print()
+        
+        if score.suggestions:
+            print("**Suggestions:**")
+            print()
+            for suggestion in score.suggestions:
+                print(f"- {suggestion}")
+            print()

package/lib/darkarts/priority_analyzer/analyzer.py

@@ -0,0 +1,301 @@
+"""
+Main Priority Analyzer for VooDocs
+
+Combines complexity, security, and dependency analysis to prioritize files.
+"""
+
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import List, Dict, Optional
+
+from .complexity import ComplexityAnalyzer
+from .security import SecurityAnalyzer
+from .dependencies import DependencyAnalyzer
+
+
+@dataclass
+class FileScore:
+    """Score and analysis for a single file."""
+    filepath: str
+    priority_score: float
+    complexity_score: int
+    security_score: int
+    dependency_score: int
+    annotation_penalty: int
+    priority_level: str
+    reasons: List[str]
+    suggestions: List[str]
+    
+    # Detailed metrics
+    loc: int = 0
+    cyclomatic: int = 0
+    functions: int = 0
+    security_keywords: List[str] = None
+    import_count: int = 0
+    dependent_count: int = 0
+    annotation_coverage: float = 0.0
+
+
+class PriorityAnalyzer:
+    """Main analyzer that combines all scoring components."""
+    
+    # Scoring weights
+    COMPLEXITY_WEIGHT = 0.30
+    SECURITY_WEIGHT = 0.40
+    DEPENDENCY_WEIGHT = 0.20
+    ANNOTATION_WEIGHT = 0.10
+    
+    # Priority level thresholds
+    PRIORITY_THRESHOLDS = {
+        'CRITICAL': 80,
+        'HIGH': 60,
+        'MEDIUM': 40,
+        'LOW': 20,
+        'MINIMAL': 0
+    }
+    
+    # Supported file extensions
+    SUPPORTED_EXTENSIONS = {'.py', '.ts', '.tsx', '.js', '.jsx', '.sol'}
+    
+    def __init__(self, project_root: str = None):
+        """
+        Initialize priority analyzer.
+        
+        Args:
+            project_root: Root directory of the project
+        """
+        self.project_root = project_root
+        self.complexity_analyzer = ComplexityAnalyzer()
+        self.security_analyzer = SecurityAnalyzer()
+        self.dependency_analyzer = DependencyAnalyzer(project_root)
+    
+    def analyze_file(self, filepath: str, all_files: List[str] = None) -> FileScore:
+        """
+        Analyze a single file and calculate priority score.
+        
+        Args:
+            filepath: Path to file to analyze
+            all_files: List of all files (for dependency analysis)
+            
+        Returns:
+            FileScore object with complete analysis
+        """
+        # Run individual analyzers
+        complexity = self.complexity_analyzer.analyze_file(filepath)
+        security = self.security_analyzer.analyze_file(filepath)
+        dependencies = self.dependency_analyzer.analyze_file(filepath, all_files)
+        
+        # Check annotation coverage
+        annotation_coverage, annotation_penalty = self._check_annotation_coverage(filepath)
+        
+        # Calculate weighted priority score
+        priority_score = (
+            complexity['total_score'] * self.COMPLEXITY_WEIGHT +
+            security['total_score'] * self.SECURITY_WEIGHT +
+            dependencies['total_score'] * self.DEPENDENCY_WEIGHT +
+            annotation_penalty * self.ANNOTATION_WEIGHT
+        )
+        
+        # Determine priority level
+        priority_level = self._get_priority_level(priority_score)
+        
+        # Generate reasons and suggestions
+        reasons = self._generate_reasons(complexity, security, dependencies, annotation_coverage)
+        suggestions = self._generate_suggestions(complexity, security, dependencies, annotation_coverage)
+        
+        # Combine all security keywords
+        all_keywords = (
+            security.get('critical_keywords', []) +
+            security.get('high_keywords', []) +
+            security.get('medium_keywords', [])
+        )
+        
+        return FileScore(
+            filepath=filepath,
+            priority_score=round(priority_score, 1),
+            complexity_score=complexity['total_score'],
+            security_score=security['total_score'],
+            dependency_score=dependencies['total_score'],
+            annotation_penalty=annotation_penalty,
+            priority_level=priority_level,
+            reasons=reasons,
+            suggestions=suggestions,
+            loc=complexity.get('loc', 0),
+            cyclomatic=complexity.get('cyclomatic', 0),
+            functions=complexity.get('functions', 0),
+            security_keywords=all_keywords[:10],  # Top 10
+            import_count=dependencies.get('import_count', 0),
+            dependent_count=dependencies.get('dependent_count', 0),
+            annotation_coverage=annotation_coverage
+        )
+    
+    def analyze_directory(self, dirpath: str, recursive: bool = True, 
+                         exclude_patterns: List[str] = None) -> List[FileScore]:
+        """
+        Analyze all files in a directory.
+        
+        Args:
+            dirpath: Directory path to analyze
+            recursive: Whether to scan subdirectories
+            exclude_patterns: Patterns to exclude (e.g., ['node_modules', 'dist'])
+            
+        Returns:
+            List of FileScore objects, sorted by priority (highest first)
+        """
+        if exclude_patterns is None:
+            exclude_patterns = ['node_modules', 'dist', 'build', '.git', '__pycache__']
+        
+        # Find all files
+        all_files = self._find_files(dirpath, recursive, exclude_patterns)
+        
+        # Analyze each file
+        scores = []
+        for filepath in all_files:
+            try:
+                score = self.analyze_file(filepath, all_files)
+                scores.append(score)
+            except Exception as e:
+                print(f"Warning: Failed to analyze {filepath}: {e}")
+        
+        # Sort by priority score (highest first)
+        scores.sort(key=lambda x: x.priority_score, reverse=True)
+        
+        return scores
+    
+    def _find_files(self, dirpath: str, recursive: bool, exclude_patterns: List[str]) -> List[str]:
+        """Find all supported files in directory."""
+        files = []
+        
+        if recursive:
+            for root, dirs, filenames in os.walk(dirpath):
+                # Filter out excluded directories
+                dirs[:] = [d for d in dirs if not any(pattern in d for pattern in exclude_patterns)]
+                
+                for filename in filenames:
+                    filepath = os.path.join(root, filename)
+                    if Path(filepath).suffix in self.SUPPORTED_EXTENSIONS:
+                        files.append(filepath)
+        else:
+            for filename in os.listdir(dirpath):
+                filepath = os.path.join(dirpath, filename)
+                if os.path.isfile(filepath) and Path(filepath).suffix in self.SUPPORTED_EXTENSIONS:
+                    files.append(filepath)
+        
+        return files
+    
+    def _check_annotation_coverage(self, filepath: str) -> tuple:
+        """
+        Check VooDocs annotation coverage in file.
+        
+        Returns:
+            Tuple of (coverage_percentage, penalty_score)
+        """
+        try:
+            with open(filepath, 'r', encoding='utf-8') as f:
+                content = f.read()
+        except:
+            return (0.0, -50)
+        
+        # Count @darkarts annotations
+        darkarts_count = len(re.findall(r'@darkarts(-lite)?', content))
+        
+        # Count total functions (rough estimate)
+        function_patterns = [
+            r'^\s*def\s+\w+\s*\(',  # Python
+            r'^\s*(export\s+)?(async\s+)?function\s+\w+\s*\(',  # JS/TS function
+            r'^\s*(const|let|var)\s+\w+\s*=\s*(\([^)]*\)|[^=]+)\s*=>',  # Arrow function
+            r'^\s*function\s+\w+\s*\(',  # Solidity
+        ]
+        
+        total_functions = 0
+        for pattern in function_patterns:
+            total_functions += len(re.findall(pattern, content, re.MULTILINE))
+        
+        # Calculate coverage
+        if total_functions == 0:
+            # No functions, check if file has any annotations
+            coverage = 1.0 if darkarts_count > 0 else 0.0
+        else:
+            coverage = min(1.0, darkarts_count / total_functions)
+        
+        # Calculate penalty
+        if coverage == 0:
+            penalty = -50
+        elif coverage < 0.5:
+            penalty = -25
+        else:
+            penalty = 0
+        
+        return (coverage, penalty)
+    
+    def _get_priority_level(self, score: float) -> str:
+        """Determine priority level from score."""
+        if score >= self.PRIORITY_THRESHOLDS['CRITICAL']:
+            return 'CRITICAL'
+        elif score >= self.PRIORITY_THRESHOLDS['HIGH']:
+            return 'HIGH'
+        elif score >= self.PRIORITY_THRESHOLDS['MEDIUM']:
+            return 'MEDIUM'
+        elif score >= self.PRIORITY_THRESHOLDS['LOW']:
+            return 'LOW'
+        else:
+            return 'MINIMAL'
+    
+    def _generate_reasons(self, complexity: Dict, security: Dict, 
+                         dependencies: Dict, annotation_coverage: float) -> List[str]:
+        """Generate human-readable reasons for priority score."""
+        reasons = []
+        
+        # Complexity reasons
+        if complexity['total_score'] >= 60:
+            reasons.append(f"High complexity: {complexity['loc']} LOC, {complexity['functions']} functions")
+        elif complexity['total_score'] >= 30:
+            reasons.append(f"Medium complexity: {complexity['loc']} LOC, {complexity['functions']} functions")
+        
+        # Security reasons
+        if security.get('critical_keywords'):
+            keywords = ', '.join(security['critical_keywords'][:5])
+            reasons.append(f"Security keywords: {keywords}")
+        
+        # Dependency reasons
+        if dependencies.get('dependent_count', 0) > 0:
+            reasons.append(f"Depended upon by {dependencies['dependent_count']} file(s)")
+        
+        # Annotation reasons
+        if annotation_coverage == 0:
+            reasons.append("No VooDocs annotations found")
+        elif annotation_coverage < 0.5:
+            reasons.append(f"Partial annotations ({int(annotation_coverage * 100)}% coverage)")
+        
+        return reasons
+    
+    def _generate_suggestions(self, complexity: Dict, security: Dict, 
+                            dependencies: Dict, annotation_coverage: float) -> List[str]:
+        """Generate actionable suggestions."""
+        suggestions = []
+        
+        # Security suggestions
+        suggestions.extend(self.security_analyzer.get_security_suggestions(security))
+        
+        # Dependency suggestions
+        suggestions.extend(self.dependency_analyzer.get_dependency_suggestions(dependencies))
+        
+        # Complexity suggestions
+        if complexity['total_score'] >= 60:
+            suggestions.append("Break down complex logic into smaller functions")
+            suggestions.append("Document high-complexity sections")
+        
+        # Annotation suggestions
+        if annotation_coverage == 0:
+            suggestions.append("Add @darkarts annotations to all functions")
+            suggestions.append("Start with module-level purpose annotation")
+        elif annotation_coverage < 1.0:
+            suggestions.append("Complete remaining function annotations")
+        
+        # Generic suggestions
+        if not suggestions:
+            suggestions.append("Add comprehensive VooDocs annotations")
+        
+        return suggestions[:5]  # Limit to top 5