solarwindpy 0.0.1.dev0__py3-none-any.whl → 0.1.1__py3-none-any.whl

plans/documentation-template-fix/3-Build-System-Integration.md

@@ -0,0 +1,766 @@
+# Phase 3: Build System Integration
+
+## Objective
+Optimize and enhance the documentation build system to seamlessly integrate with the enhanced templates, ensuring robust automated processing and CI/CD compatibility.
+
+## Current Build System Analysis
+
+### Build Pipeline Overview
+```
+sphinx-apidoc → add_no_index.py → Sphinx HTML Build → Deployment
+     ↓              ↓                    ↓              ↓
+  RST Generation  Post-processing    HTML Generation   Publishing
+```
+
+### Current Components
+
+#### 3.1 Makefile Integration (`docs/Makefile`)
+
+**Current Build Targets**:
+```makefile
+# Line 15: HTML build depends on API generation
+html: api
+
+# Lines 18-22: API generation with post-processing  
+api:
+	@echo "Generating API documentation..."
+	sphinx-apidoc -f -o $(SOURCEDIR)/api ../../solarwindpy/solarwindpy --separate
+	@echo "Post-processing API files..."
+	python add_no_index.py
+```
+
+**Analysis**:
+- ✅ **Dependency management**: HTML build properly depends on API generation
+- ✅ **Post-processing integration**: Automated post-processing step
+- ✅ **Clean separation**: API generation separate from HTML build
+- ❌ **No template validation**: No verification that templates are working
+- ❌ **No physics content validation**: No checking of enhanced content
+
+#### 3.2 Post-Processing Script (`docs/add_no_index.py`)
+
+**Current Functionality**:
+```python
+# Lines 30-43: Add :no-index: to automodule directives
+pattern = r'(\.\. automodule:: .+?)(\n   :members:)'
+replacement = r'\1\n   :no-index:\2'
+
+for file_path in glob.glob('source/api/*.rst'):
+    # Process each generated RST file
+    content = re.sub(pattern, replacement, content)
+```
+
+**Analysis**:
+- ✅ **Functional**: Successfully adds :no-index: directives
+- ✅ **Comprehensive**: Processes all API files
+- ✅ **Reliable**: Handles edge cases correctly
+- ❌ **Limited scope**: Only handles :no-index: addition
+- ❌ **No template-specific processing**: Doesn't handle enhanced template content
+
+#### 3.3 CI/CD Integration (`.github/workflows/docs.yml`)
+
+**Current Workflow**:
+```yaml
+# Lines 54-55: Full clean build
+- name: Build documentation
+  run: |
+    cd docs
+    make clean
+    make html
+
+# Lines 105-106: Deployment build
+- name: Deploy documentation  
+  run: |
+    cd docs
+    make clean && make html
+```
+
+**Analysis**:
+- ✅ **Clean builds**: Ensures fresh documentation generation
+- ✅ **Dependency installation**: Proper Python environment setup
+- ✅ **Artifact management**: Handles build outputs correctly
+- ❌ **No validation steps**: Missing template and content validation
+- ❌ **No performance monitoring**: No build time or size tracking
+
+## Enhanced Build System Design
+
+### 3.1 Enhanced Post-Processing Framework
+
+#### Enhanced `add_no_index.py` Script
+
+**Current Limitations**:
+- Only processes :no-index: directives
+- No validation of template-generated content
+- No physics-specific content enhancement
+- No error reporting for template issues
+
+**Enhanced Script Design**:
+
+```python
+#!/usr/bin/env python3
+"""
+Enhanced post-processing script for SolarWindPy documentation.
+Handles template-generated content validation and physics-specific enhancements.
+"""
+
+import os
+import re
+import glob
+import sys
+from pathlib import Path
+from typing import List, Dict, Tuple
+
+class DocumentationProcessor:
+    """Enhanced documentation post-processor."""
+    
+    def __init__(self, source_dir: str = "source/api"):
+        self.source_dir = Path(source_dir)
+        self.errors: List[str] = []
+        self.warnings: List[str] = []
+        self.stats: Dict[str, int] = {
+            'files_processed': 0,
+            'physics_sections_added': 0,
+            'cross_references_fixed': 0,
+            'validation_warnings': 0
+        }
+    
+    def process_no_index_directives(self, content: str) -> str:
+        """Add :no-index: directives to automodule declarations."""
+        pattern = r'(\.\. automodule:: .+?)(\n   :members:)'
+        replacement = r'\1\n   :no-index:\2'
+        return re.sub(pattern, replacement, content)
+    
+    def validate_physics_sections(self, content: str, filename: str) -> str:
+        """Validate and enhance physics-specific sections."""
+        # Check for physics classes that should have enhanced sections
+        physics_classes = ['Plasma', 'Ion', 'Base']
+        
+        for physics_class in physics_classes:
+            if f'autoclass:: solarwindpy.core.{physics_class.lower()}' in content:
+                if 'Physical Properties' not in content:
+                    self.warnings.append(f"{filename}: Missing Physical Properties section for {physics_class}")
+                    self.stats['validation_warnings'] += 1
+                else:
+                    self.stats['physics_sections_added'] += 1
+        
+        return content
+    
+    def fix_cross_references(self, content: str) -> str:
+        """Fix and enhance cross-references for physics concepts."""
+        # Fix common cross-reference patterns
+        cross_ref_fixes = {
+            r'solarwindpy\.core\.plasma\.Plasma': r':py:class:`~solarwindpy.core.plasma.Plasma`',
+            r'solarwindpy\.core\.ions\.Ion': r':py:class:`~solarwindpy.core.ions.Ion`',
+            r'solarwindpy\.core\.base\.Base': r':py:class:`~solarwindpy.core.base.Base`',
+        }
+        
+        for pattern, replacement in cross_ref_fixes.items():
+            if re.search(pattern, content) and not re.search(replacement.replace('\\', ''), content):
+                content = re.sub(pattern, replacement, content)
+                self.stats['cross_references_fixed'] += 1
+        
+        return content
+    
+    def process_file(self, file_path: Path) -> None:
+        """Process a single RST file."""
+        try:
+            with open(file_path, 'r', encoding='utf-8') as f:
+                content = f.read()
+            
+            # Apply all processing steps
+            content = self.process_no_index_directives(content)
+            content = self.validate_physics_sections(content, file_path.name)
+            content = self.fix_cross_references(content)
+            
+            with open(file_path, 'w', encoding='utf-8') as f:
+                f.write(content)
+            
+            self.stats['files_processed'] += 1
+            
+        except Exception as e:
+            error_msg = f"Error processing {file_path}: {e}"
+            self.errors.append(error_msg)
+            print(f"❌ {error_msg}", file=sys.stderr)
+    
+    def process_all_files(self) -> bool:
+        """Process all RST files in the API directory."""
+        rst_files = list(self.source_dir.glob("*.rst"))
+        
+        if not rst_files:
+            self.errors.append(f"No RST files found in {self.source_dir}")
+            return False
+        
+        print(f"🔄 Processing {len(rst_files)} documentation files...")
+        
+        for rst_file in rst_files:
+            self.process_file(rst_file)
+        
+        return len(self.errors) == 0
+    
+    def print_summary(self) -> None:
+        """Print processing summary."""
+        print(f"\n📊 Documentation Processing Summary:")
+        print(f"   • Files processed: {self.stats['files_processed']}")
+        print(f"   • Physics sections validated: {self.stats['physics_sections_added']}")
+        print(f"   • Cross-references fixed: {self.stats['cross_references_fixed']}")
+        print(f"   • Warnings: {self.stats['validation_warnings']}")
+        print(f"   • Errors: {len(self.errors)}")
+        
+        if self.warnings:
+            print(f"\n⚠️  Warnings:")
+            for warning in self.warnings:
+                print(f"   • {warning}")
+        
+        if self.errors:
+            print(f"\n❌ Errors:")
+            for error in self.errors:
+                print(f"   • {error}")
+
+def main():
+    """Main processing function."""
+    processor = DocumentationProcessor()
+    
+    success = processor.process_all_files()
+    processor.print_summary()
+    
+    if success:
+        print("\n✅ Documentation processing completed successfully!")
+        return 0
+    else:
+        print("\n❌ Documentation processing failed!")
+        return 1
+
+if __name__ == "__main__":
+    sys.exit(main())
+```
+
+**Enhancement Benefits**:
+1. **Comprehensive Processing**: Handles multiple post-processing tasks
+2. **Physics Validation**: Validates template-generated physics content
+3. **Cross-reference Fixing**: Automatically fixes broken references
+4. **Error Reporting**: Detailed error and warning reporting
+5. **Statistics Tracking**: Provides build metrics and insights
+
+### 3.2 Build System Validation Framework
+
+#### Template Validation Integration
+
+**Build Target Enhancement**:
+
+```makefile
+# Enhanced Makefile with validation
+.PHONY: validate-templates
+validate-templates:
+	@echo "🔍 Validating documentation templates..."
+	python validate_templates.py
+	@echo "✅ Template validation complete"
+
+.PHONY: api-enhanced
+api-enhanced: validate-templates
+	@echo "🔧 Generating enhanced API documentation..."
+	sphinx-apidoc -f -o $(SOURCEDIR)/api ../../solarwindpy/solarwindpy --separate
+	@echo "⚙️  Post-processing with enhanced framework..."
+	python add_no_index.py
+	@echo "🔍 Validating generated documentation..."
+	python validate_generated_docs.py
+	@echo "✅ Enhanced API generation complete"
+
+# Update HTML target to use enhanced API
+html: api-enhanced
+	@echo "🏗️  Building HTML documentation..."
+	$(SPHINXBUILD) -b html $(SOURCEDIR) $(BUILDDIR)/html $(SPHINXOPTS)
+	@echo "✅ HTML documentation build complete"
+```
+
+#### Generated Documentation Validator
+
+**`docs/validate_generated_docs.py`**:
+
+```python
+#!/usr/bin/env python3
+"""
+Validation script for generated documentation content.
+Ensures template enhancements are properly applied.
+"""
+
+import os
+import re
+import sys
+from pathlib import Path
+from typing import List, Dict
+
+class GeneratedDocValidator:
+    """Validator for template-generated documentation."""
+    
+    def __init__(self, api_dir: str = "source/api"):
+        self.api_dir = Path(api_dir)
+        self.validation_results: Dict[str, List[str]] = {
+            'passed': [],
+            'warnings': [],
+            'failed': []
+        }
+    
+    def validate_physics_sections(self, file_path: Path) -> bool:
+        """Validate physics-specific sections in class documentation."""
+        with open(file_path, 'r') as f:
+            content = f.read()
+        
+        # Check for core physics classes
+        physics_classes = ['plasma.rst', 'ions.rst', 'base.rst']
+        
+        if any(cls in file_path.name for cls in physics_classes):
+            required_sections = ['Physical Properties', 'Units and Dimensions']
+            missing_sections = []
+            
+            for section in required_sections:
+                if section not in content:
+                    missing_sections.append(section)
+            
+            if missing_sections:
+                self.validation_results['warnings'].append(
+                    f"{file_path.name}: Missing sections: {', '.join(missing_sections)}"
+                )
+                return False
+            else:
+                self.validation_results['passed'].append(
+                    f"{file_path.name}: All physics sections present"
+                )
+                return True
+        
+        return True
+    
+    def validate_cross_references(self, file_path: Path) -> bool:
+        """Validate cross-references are properly formatted."""
+        with open(file_path, 'r') as f:
+            content = f.read()
+        
+        # Look for broken reference patterns
+        broken_patterns = [
+            r'solarwindpy\.[a-zA-Z.]+[A-Z][a-zA-Z]+',  # Unformatted class references
+            r'\.\. autoclass:: [^\n]*\n(?!\s*:)',      # Missing options
+        ]
+        
+        issues = []
+        for i, pattern in enumerate(broken_patterns):
+            matches = re.findall(pattern, content)
+            if matches:
+                issues.extend(matches)
+        
+        if issues:
+            self.validation_results['warnings'].append(
+                f"{file_path.name}: Potential reference issues: {len(issues)}"
+            )
+            return False
+        
+        return True
+    
+    def validate_template_application(self, file_path: Path) -> bool:
+        """Validate that templates were properly applied."""
+        with open(file_path, 'r') as f:
+            content = f.read()
+        
+        # Check for template artifacts that indicate proper processing
+        template_indicators = [':no-index:', 'rubric::']
+        
+        has_indicators = any(indicator in content for indicator in template_indicators)
+        
+        if not has_indicators:
+            self.validation_results['failed'].append(
+                f"{file_path.name}: No template processing indicators found"
+            )
+            return False
+        
+        return True
+    
+    def validate_all_files(self) -> bool:
+        """Validate all generated documentation files."""
+        rst_files = list(self.api_dir.glob("*.rst"))
+        
+        if not rst_files:
+            self.validation_results['failed'].append("No RST files found for validation")
+            return False
+        
+        print(f"🔍 Validating {len(rst_files)} generated documentation files...")
+        
+        all_valid = True
+        for rst_file in rst_files:
+            file_valid = True
+            file_valid &= self.validate_physics_sections(rst_file)
+            file_valid &= self.validate_cross_references(rst_file)
+            file_valid &= self.validate_template_application(rst_file)
+            
+            if not file_valid:
+                all_valid = False
+        
+        return all_valid
+    
+    def print_results(self) -> None:
+        """Print validation results."""
+        print(f"\n📊 Documentation Validation Results:")
+        print(f"   ✅ Passed: {len(self.validation_results['passed'])}")
+        print(f"   ⚠️  Warnings: {len(self.validation_results['warnings'])}")
+        print(f"   ❌ Failed: {len(self.validation_results['failed'])}")
+        
+        for category, messages in self.validation_results.items():
+            if messages and category != 'passed':
+                print(f"\n{category.title()}:")
+                for message in messages:
+                    print(f"   • {message}")
+
+def main():
+    """Main validation function."""
+    validator = GeneratedDocValidator()
+    
+    success = validator.validate_all_files()
+    validator.print_results()
+    
+    if success:
+        print("\n✅ All documentation validation checks passed!")
+        return 0
+    else:
+        print("\n⚠️  Some validation issues found (see details above)")
+        return 0  # Don't fail build for warnings, only for critical errors
+
+if __name__ == "__main__":
+    sys.exit(main())
+```
+
+### 3.3 Performance Optimization
+
+#### Build Time Monitoring
+
+**Build Performance Script** (`docs/monitor_build.py`):
+
+```python
+#!/usr/bin/env python3
+"""
+Build performance monitoring for documentation system.
+Tracks build times and identifies bottlenecks.
+"""
+
+import time
+import subprocess
+import sys
+from pathlib import Path
+
+class BuildMonitor:
+    """Monitor documentation build performance."""
+    
+    def __init__(self):
+        self.timings = {}
+        self.start_time = None
+    
+    def start_timer(self, operation: str):
+        """Start timing an operation."""
+        self.start_time = time.time()
+        print(f"⏱️  Starting {operation}...")
+    
+    def end_timer(self, operation: str):
+        """End timing and record result."""
+        if self.start_time:
+            duration = time.time() - self.start_time
+            self.timings[operation] = duration
+            print(f"✅ {operation} completed in {duration:.2f}s")
+            self.start_time = None
+    
+    def run_timed_command(self, command: list, operation: str):
+        """Run a command with timing."""
+        self.start_timer(operation)
+        try:
+            result = subprocess.run(command, check=True, capture_output=True, text=True)
+            self.end_timer(operation)
+            return result
+        except subprocess.CalledProcessError as e:
+            print(f"❌ {operation} failed: {e}")
+            return None
+    
+    def print_summary(self):
+        """Print build performance summary."""
+        if not self.timings:
+            print("No timing data collected")
+            return
+        
+        total_time = sum(self.timings.values())
+        
+        print(f"\n📊 Build Performance Summary:")
+        print(f"   Total build time: {total_time:.2f}s")
+        print(f"   Individual operations:")
+        
+        for operation, duration in sorted(self.timings.items(), key=lambda x: x[1], reverse=True):
+            percentage = (duration / total_time) * 100
+            print(f"     • {operation}: {duration:.2f}s ({percentage:.1f}%)")
+
+def main():
+    """Monitor a full documentation build."""
+    monitor = BuildMonitor()
+    
+    # Change to docs directory
+    os.chdir('docs')
+    
+    # Clean build
+    monitor.run_timed_command(['make', 'clean'], 'Clean build directory')
+    
+    # Template validation
+    monitor.run_timed_command(['python', 'validate_templates.py'], 'Template validation')
+    
+    # API generation
+    monitor.run_timed_command(['make', 'api-enhanced'], 'Enhanced API generation')
+    
+    # HTML build
+    monitor.run_timed_command(['make', 'html'], 'HTML generation')
+    
+    # Print results
+    monitor.print_summary()
+    
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())
+```
+
+### 3.4 CI/CD Integration Enhancement
+
+#### Enhanced GitHub Actions Workflow
+
+**Updated `.github/workflows/docs.yml`**:
+
+```yaml
+name: Documentation Build and Deploy
+
+on:
+  push:
+    branches: [main, master]
+  pull_request:
+    branches: [main, master]
+
+jobs:
+  build-docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev.txt
+          pip install -r docs/requirements.txt
+          pip install -e .
+      
+      # Template validation step
+      - name: Validate documentation templates
+        run: |
+          cd docs
+          python validate_templates.py
+      
+      # Enhanced build with monitoring
+      - name: Build documentation with monitoring
+        run: |
+          cd docs
+          python monitor_build.py
+      
+      # Validation of generated docs
+      - name: Validate generated documentation
+        run: |
+          cd docs
+          python validate_generated_docs.py
+      
+      # Check for build warnings
+      - name: Check for Sphinx warnings
+        run: |
+          cd docs
+          make html 2>&1 | tee build.log
+          if grep -q "WARNING" build.log; then
+            echo "⚠️  Sphinx warnings found:"
+            grep "WARNING" build.log
+          else
+            echo "✅ No Sphinx warnings"
+          fi
+      
+      - name: Upload documentation artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: documentation
+          path: docs/_build/html/
+          retention-days: 30
+      
+      - name: Upload build logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: build-logs
+          path: docs/build.log
+          retention-days: 7
+```
+
+**Enhancement Benefits**:
+1. **Template Validation**: Catches template issues before build
+2. **Performance Monitoring**: Tracks build performance over time
+3. **Generated Content Validation**: Ensures template enhancements work
+4. **Warning Detection**: Identifies Sphinx warnings automatically
+5. **Comprehensive Artifacts**: Saves both docs and build logs
+
+### 3.5 Local Development Integration
+
+#### Enhanced Development Makefile Targets
+
+**Additional Makefile Targets**:
+
+```makefile
+# Development-specific targets for enhanced build system
+
+.PHONY: dev-build
+dev-build: validate-templates api-enhanced
+	@echo "🔧 Development build with full validation..."
+	$(SPHINXBUILD) -b html $(SOURCEDIR) $(BUILDDIR)/html $(SPHINXOPTS) -W
+	@echo "🌐 Opening documentation in browser..."
+	python -c "import webbrowser; webbrowser.open('file://$(PWD)/_build/html/index.html')"
+
+.PHONY: fast-build  
+fast-build:
+	@echo "⚡ Fast build (skip validation)..."
+	sphinx-apidoc -f -o $(SOURCEDIR)/api ../../solarwindpy/solarwindpy --separate
+	python add_no_index.py
+	$(SPHINXBUILD) -b html $(SOURCEDIR) $(BUILDDIR)/html $(SPHINXOPTS)
+
+.PHONY: monitor-build
+monitor-build:
+	@echo "📊 Building with performance monitoring..."
+	python monitor_build.py
+
+.PHONY: validate-all
+validate-all: validate-templates
+	@echo "🔍 Comprehensive validation..."
+	make api-enhanced
+	python validate_generated_docs.py
+	@echo "✅ All validation checks complete"
+
+# Watch for template changes and rebuild
+.PHONY: watch-templates
+watch-templates:
+	@echo "👀 Watching templates for changes..."
+	@echo "Run 'make dev-build' in another terminal when templates change"
+	python -c "
+import time
+from pathlib import Path
+from watchdog.observers import Observer
+from watchdog.events import FileSystemEventHandler
+
+class TemplateHandler(FileSystemEventHandler):
+    def on_modified(self, event):
+        if event.src_path.endswith('.rst'):
+            print(f'🔄 Template changed: {event.src_path}')
+            print('   Run: make dev-build')
+
+observer = Observer()
+observer.schedule(TemplateHandler(), 'source/_templates', recursive=True)
+observer.start()
+
+try:
+    while True:
+        time.sleep(1)
+except KeyboardInterrupt:
+    observer.stop()
+observer.join()
+"
+```
+
+## Success Criteria
+
+### Build System Integration Validation
+
+- [ ] **Enhanced post-processing** handles template-generated content correctly
+- [ ] **Template validation** catches syntax errors before build
+- [ ] **Generated content validation** ensures physics sections are present
+- [ ] **Performance monitoring** provides build time insights
+- [ ] **CI/CD integration** works seamlessly with enhancements
+- [ ] **Local development** workflow is improved and streamlined
+
+### Quality Assurance
+
+- [ ] **No build failures** introduced by integration changes
+- [ ] **No regression** in existing documentation quality
+- [ ] **Enhanced physics content** appears in generated documentation
+- [ ] **Cross-references work** correctly after processing
+- [ ] **Warning-free builds** for all documentation
+- [ ] **Fast build times** maintained despite enhancements
+
+### Developer Experience
+
+- [ ] **Clear error messages** when template issues occur
+- [ ] **Automated validation** prevents common mistakes
+- [ ] **Performance feedback** helps optimize build process
+- [ ] **Development targets** streamline local workflow
+- [ ] **Comprehensive logging** aids in debugging issues
+
+## Implementation Timeline
+
+| Task | Duration | Dependencies | Validation |
+|------|----------|--------------|------------|
+| **Enhanced post-processing script** | 60 min | Phase 2 templates | Script testing |
+| **Validation framework creation** | 45 min | Enhanced script | Framework testing |
+| **Makefile target enhancements** | 30 min | Validation framework | Build testing |
+| **Performance monitoring setup** | 30 min | Makefile changes | Performance validation |
+| **CI/CD workflow updates** | 30 min | All components | Workflow testing |
+| **Local development integration** | 30 min | CI/CD changes | Developer testing |
+
+**Total Phase 3 Time**: 3.5 hours
+
+## Commit Tracking
+
+- Enhanced post-processing: `<checksum_enhanced_processing>`
+- Validation framework: `<checksum_validation_framework>`
+- Makefile enhancements: `<checksum_makefile_enhanced>`
+- Performance monitoring: `<checksum_performance_monitoring>`
+- CI/CD integration: `<checksum_cicd_integration>`
+- Local development tools: `<checksum_local_dev_tools>`
+- Phase 3 completion: `<checksum_phase3_complete>`
+
+## Risk Mitigation
+
+### Integration Risks
+
+| Risk | Probability | Impact | Mitigation |
+|------|-------------|--------|------------|
+| **Build system breakage** | Low | High | Incremental testing + rollback plan |
+| **Performance degradation** | Medium | Medium | Performance monitoring + optimization |
+| **CI/CD pipeline failures** | Low | High | Staged deployment + validation |
+| **Developer workflow disruption** | Medium | Low | Clear documentation + training |
+
+### Rollback Strategy
+
+```bash
+# Emergency rollback for build system issues
+
+# 1. Restore original post-processing
+git checkout HEAD~1 -- docs/add_no_index.py
+
+# 2. Restore original Makefile  
+git checkout HEAD~1 -- docs/Makefile
+
+# 3. Restore original CI/CD workflow
+git checkout HEAD~1 -- .github/workflows/docs.yml
+
+# 4. Test basic build
+cd docs
+make clean
+make html
+
+# 5. If successful, commit rollback
+git add . && git commit -m "Emergency rollback of build system changes"
+```
+
+## Next Phase Preparation
+
+Phase 4 (Testing & Validation) should focus on:
+1. **Comprehensive build testing** across different environments
+2. **Template content validation** for scientific accuracy
+3. **Performance regression testing** to ensure efficiency
+4. **CI/CD integration testing** with multiple scenarios
+5. **User acceptance testing** with physics documentation users
+
+The enhanced build system provides robust infrastructure for persistent, high-quality physics documentation with comprehensive validation and monitoring capabilities.