rust-crate-pipeline 1.4.0__tar.gz → 1.5.2__tar.gz

{rust_crate_pipeline-1.4.0 → rust_crate_pipeline-1.5.2}/CHANGELOG.md

@@ -2,6 +2,101 @@
 
 All notable changes to the Rust Crate Pipeline project.
 
+## [1.5.2] - 2025-06-20
+
+### 🎯 Compliance, Test, and Build Fixes
+
+#### ✨ Improvements
+- **Rule Zero Compliance**: Achieved full compliance with Rule Zero principles across all modules
+- **PEP8 Compliance**: Resolved all PEP8 violations, ensuring adherence to Python coding standards
+- **Type and Interface Fixes**: Applied all necessary fixes for type and interface propagation
+- **Test Coverage**: Enhanced test coverage with robust default and test configurations for all pipelines
+- **Async Test Support**: Integrated support for async tests using pytest-asyncio
+
+#### 🔧 Technical Updates
+- **Build Validation**: Thorough validation of the build process, ensuring readiness for production
+- **Dependency Updates**: Updated dependencies to latest compatible versions
+- **Configuration Refinements**: Minor refinements to configuration files for consistency
+
+#### 📝 Documentation
+- **README Updates**: Minor updates to README.md to reflect recent changes
+- **CLI Documentation**: Ensured command-line options table is up-to-date
+- **Configuration Examples**: Reviewed and updated JSON configuration file examples
+
+#### ⚖️ Rule Zero Methods Applied
+- **Alignment**: All configurations now consistently align with production environment standards
+- **Validation**: Enhanced test coverage ensures configuration consistency across all modules
+- **Transparency**: Clear documentation of model path requirements and configuration options
+- **Adaptability**: Modular configuration system supports easy adaptation to different model paths
+
+## [1.5.1] - 2025-06-20
+
+### 🔧 Configuration Standardization & Rule Zero Alignment
+
+#### ✨ Improvements
+- **Model Path Consistency**: Standardized all configuration files, CLI defaults, and documentation to use proper GGUF model paths (`~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf`)
+- **Rule Zero Compliance**: Enhanced alignment with Rule Zero principles for transparency, validation, and adaptability
+- **Documentation Coherence**: Comprehensive updates across README.md, CLI help text, and configuration examples
+- **Test Standardization**: Updated all test files to use consistent GGUF model path references
+
+#### 🔧 Technical Updates
+- **CLI Consistency**: Updated `--crawl4ai-model` default value and help text to reflect correct GGUF paths
+- **Configuration Files**: Ensured JSON configuration examples use proper model path format
+- **Test Coverage**: Updated integration and demo tests to use standardized model paths
+- **Code Quality**: Removed inconsistent Ollama references in favor of llama-cpp-python approach
+
+#### 📝 Documentation
+- **README Updates**: Corrected all usage examples to show proper GGUF model configuration
+- **CLI Documentation**: Updated command-line options table with accurate default values
+- **Configuration Examples**: Standardized JSON configuration file examples
+- **Badge Updates**: Updated version badges and PyPI references to v1.5.1
+
+#### ⚖️ Rule Zero Methods Applied
+- **Alignment**: All configurations now consistently align with production environment standards
+- **Validation**: Enhanced test coverage ensures configuration consistency across all modules
+- **Transparency**: Clear documentation of model path requirements and configuration options
+- **Adaptability**: Modular configuration system supports easy adaptation to different model paths
+
+## [1.5.0] - 2025-06-20
+
+### 🚀 Major Release: Enhanced Web Scraping with Crawl4AI Integration
+
+#### ✨ New Features
+- **Advanced Web Scraping**: Full integration of Crawl4AI for enterprise-grade content extraction
+- **JavaScript Rendering**: Playwright-powered browser automation for dynamic content scraping
+- **LLM-Enhanced Parsing**: AI-powered README and documentation analysis
+- **Structured Data Extraction**: Intelligent parsing of docs.rs and technical documentation
+- **Quality Scoring**: Automated content quality assessment and validation
+- **Async Processing**: High-performance async web scraping with concurrent request handling
+
+#### 🔧 Enhanced Configuration
+- **New CLI Options**: 
+  - `--enable-crawl4ai`: Enable advanced web scraping (default: enabled)
+  - `--disable-crawl4ai`: Use basic scraping only
+  - `--crawl4ai-model`: Configure GGUF model path for content analysis
+- **Configuration Parameters**:
+  - `enable_crawl4ai: bool = True`
+  - `crawl4ai_model: str = "~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf"`
+  - `crawl4ai_timeout: int = 30`
+
+#### 🛡️ Reliability & Fallbacks
+- **Graceful Degradation**: Automatic fallback to basic scraping when Crawl4AI unavailable
+- **Error Handling**: Comprehensive exception management for web scraping failures
+- **Browser Management**: Automated Playwright browser installation and management
+- **Network Resilience**: Retry logic and timeout handling for web requests
+
+#### 📋 Pipeline Integration
+- **Standard Pipeline**: Full Crawl4AI support in `CrateDataPipeline`
+- **Sigil Protocol**: Enhanced scraping integrated with Rule Zero compliance
+- **Dual Mode Operation**: Seamless switching between enhanced and basic scraping
+- **Test Coverage**: Comprehensive test suite for all Crawl4AI features
+
+#### 🎯 Rule Zero Compliance
+- **Transparency**: Full audit trails for all web scraping operations
+- **Validation**: Quality scoring and content verification
+- **Alignment**: Consistent with established architecture patterns
+- **Adaptability**: Modular design with configurable scraping strategies
+
 ## [1.4.0] - 2025-06-20
 
 ### 🏆 Major Release: Rule Zero Compliance Audit Complete

rust_crate_pipeline-1.5.2/COMMIT_MESSAGE.md

@@ -0,0 +1,73 @@
+# v1.5.1: Configuration Standardization & Rule Zero Alignment
+
+## Summary
+Increment version to 1.5.1 with comprehensive standardization of model path configuration across all components, enhanced Rule Zero compliance, and documentation consistency improvements.
+
+## Changes Made
+
+### 🔧 Version Updates
+- **pyproject.toml**: Incremented version from 1.5.0 → 1.5.1
+- **setup.py**: Updated version string to 1.5.1
+- **rust_crate_pipeline/version.py**: Updated __version__ and added v1.5.1 changelog entry
+- **README.md**: Updated PyPI badge and "New in v1.5.1" announcement
+
+### 🎯 Configuration Standardization
+- **Model Path Consistency**: Standardized all references to use `~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf`
+- **CLI Defaults**: Updated `--crawl4ai-model` default value in main.py
+- **Test Files**: Updated all test configurations to use consistent GGUF model paths
+- **Documentation**: Ensured README examples and CLI table reflect correct paths
+
+### 📝 Documentation Updates
+- **README.md**: 
+  - Fixed corrupted header line
+  - Added v1.5.1 section to Recent Updates
+  - Updated version announcements and PyPI references
+  - Maintained consistency in all code examples
+- **CHANGELOG.md**: Added comprehensive v1.5.1 section detailing all changes
+- **CLI Help**: Ensured all help text shows correct default model paths
+
+### ⚖️ Rule Zero Compliance Enhancements
+- **Alignment**: All configurations now consistently align with production standards
+- **Validation**: Enhanced test coverage ensures configuration consistency
+- **Transparency**: Clear documentation of model path requirements
+- **Adaptability**: Maintained modular configuration system
+
+### 🧪 Test Improvements
+- **tests/test_crawl4ai_demo.py**: Updated model path references
+- **tests/test_crawl4ai_integration.py**: Standardized configuration examples
+- **Consistent Test Coverage**: All tests now use proper GGUF model paths
+
+## Files Modified
+- `pyproject.toml`
+- `setup.py`
+- `rust_crate_pipeline/version.py`
+- `rust_crate_pipeline/main.py`
+- `enhanced_scraping.py`
+- `README.md`
+- `CHANGELOG.md`
+- `tests/test_crawl4ai_demo.py`
+- `tests/test_crawl4ai_integration.py`
+
+## Validation
+- All version strings updated consistently across project
+- CLI help output shows correct default model paths
+- Documentation examples reflect proper GGUF configuration
+- Test files use standardized model path references
+- CHANGELOG and README properly updated for v1.5.1
+
+## Rule Zero Principles Applied
+1. **Alignment**: Standardized configuration aligns with production environment
+2. **Validation**: Enhanced test coverage validates configuration consistency
+3. **Transparency**: Clear documentation of all model path requirements
+4. **Adaptability**: Maintained flexible configuration system architecture
+
+## Impact
+- Enhanced user experience with consistent configuration
+- Improved documentation clarity and accuracy
+- Better alignment with production deployment practices
+- Stronger Rule Zero compliance across all components
+
+## Next Steps
+- Ready for git commit and tag creation
+- Documentation is production-ready
+- All configuration examples are accurate and validated

rust_crate_pipeline-1.5.2/DOCKER_DEPLOYMENT.md

@@ -0,0 +1,273 @@
+# Docker Deployment Guide for SigilDERG-Data_Production v1.5.1
+
+## Overview
+
+This guide covers deploying SigilDERG-Data_Production v1.5.1 using Docker with full Crawl4AI integration and GGUF model support.
+
+## Prerequisites
+
+- Docker Engine 20.10+ 
+- Docker Compose 2.0+
+- At least 8GB RAM available for the container
+- 4 CPU cores recommended
+- GGUF model file: `deepseek-coder-6.7b-instruct.Q4_K_M.gguf`
+
+## Model Setup
+
+### Local Model Directory
+```bash
+# Create local models directory
+mkdir -p ~/models/deepseek
+
+# Download the GGUF model (example)
+wget -O ~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf \
+  "https://example.com/path/to/model"
+```
+
+### Windows Model Directory
+```powershell
+# Create local models directory
+New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\models\deepseek"
+
+# Place your GGUF model file in:
+# %USERPROFILE%\models\deepseek\deepseek-coder-6.7b-instruct.Q4_K_M.gguf
+```
+
+## Environment Variables
+
+Create a `.env` file in the project root:
+
+```bash
+# GitHub API Token (optional but recommended)
+GITHUB_TOKEN=your_github_token_here
+
+# Logging configuration
+LOG_LEVEL=INFO
+
+# Model configuration (GGUF with llama-cpp-python)  
+MODEL_PATH=/app/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf
+LLM_MODEL_PATH=/app/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf
+CRAWL4AI_MODEL=/app/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf
+
+# LLM inference parameters
+LLM_CONTEXT_SIZE=4096
+LLM_MAX_TOKENS=512
+LLM_TEMPERATURE=0.1
+
+# Host model directory (adjust path as needed)
+# Linux/Mac: HOME=/home/username or /Users/username
+# Windows: HOME=C:/Users/username
+HOME=/path/to/your/home/directory
+```
+
+## Deployment Methods
+
+### Method 1: Docker Compose (Recommended)
+
+```bash
+# Clone the repository
+git clone https://github.com/Superuser666-Sigil/SigilDERG-Data_Production.git
+cd SigilDERG-Data_Production
+
+# Create required directories
+mkdir -p output logs cache data
+
+# Start the service
+docker-compose up -d
+
+# View logs
+docker-compose logs -f rust-crate-pipeline
+
+# Stop the service
+docker-compose down
+```
+
+### Method 2: Docker Build and Run
+
+```bash
+# Build the image
+docker build -t rust-crate-pipeline:1.5.1 .
+
+# Run the container
+docker run -d \
+  --name rust-pipeline \
+  --restart unless-stopped \
+  -v $(pwd)/output:/app/output \
+  -v $(pwd)/logs:/app/logs \
+  -v $(pwd)/cache:/app/cache \
+  -v ~/models:/app/models:ro \
+  -e GITHUB_TOKEN="${GITHUB_TOKEN}" \
+  -e LOG_LEVEL=INFO \
+  rust-crate-pipeline:1.5.1 \
+  --limit 1000 --batch-size 10
+```
+
+## Container Management
+
+### Interactive Shell Access
+```bash
+# Access running container
+docker exec -it rust-pipeline bash
+
+# Or start in interactive mode
+docker run -it --rm rust-crate-pipeline:1.5.1 bash
+```
+
+### Health Check
+```bash
+# Check container health
+docker ps
+docker inspect rust-pipeline | grep -A 10 Health
+
+# Manual health check
+docker exec rust-pipeline python -c "
+import rust_crate_pipeline
+from rust_crate_pipeline.config import PipelineConfig
+PipelineConfig()
+print('✅ Container health check passed')
+"
+```
+
+### Container Testing
+```bash
+# Run container test mode
+docker run --rm rust-crate-pipeline:1.5.1 test
+```
+
+## Configuration Validation
+
+### Verify Model Paths
+```bash
+docker exec rust-pipeline ls -la /app/models/deepseek/
+docker exec rust-pipeline python -c "
+import os
+model_path = os.environ.get('LLM_MODEL_PATH')
+print(f'Model path: {model_path}')
+print(f'Model exists: {os.path.exists(model_path) if model_path else False}')
+"
+```
+
+### Verify Crawl4AI Integration
+```bash
+docker exec rust-pipeline python -c "
+import crawl4ai
+from crawl4ai import AsyncWebCrawler
+print('✅ Crawl4AI available')
+print(f'Chromium path: /usr/bin/chromium')
+import os
+print(f'Chromium exists: {os.path.exists(\"/usr/bin/chromium\")}')
+"
+```
+
+## Log Monitoring
+
+### Using Docker Logs
+```bash
+# Follow logs
+docker logs -f rust-pipeline
+
+# View recent logs
+docker logs --tail 100 rust-pipeline
+```
+
+### Using Dozzle (Web UI)
+```bash
+# Start with monitoring profile
+docker-compose --profile monitoring up -d
+
+# Access logs at http://localhost:8081
+```
+
+## Performance Tuning
+
+### Resource Limits
+The default configuration allocates:
+- **CPU**: 4 cores limit, 2 cores reserved
+- **Memory**: 8GB limit, 4GB reserved
+
+Adjust in `docker-compose.yml`:
+```yaml
+deploy:
+  resources:
+    limits:
+      cpus: '6.0'        # Increase for better performance
+      memory: 12G        # Increase for larger models
+    reservations:
+      cpus: '3.0'
+      memory: 6G
+```
+
+### Model Optimization
+- Use GGUF models for better memory efficiency
+- Adjust `LLM_CONTEXT_SIZE` based on available memory
+- Lower `LLM_TEMPERATURE` for more deterministic results
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Model not found**
+   ```bash
+   # Check model mount and permissions
+   docker exec rust-pipeline ls -la /app/models/deepseek/
+   docker exec rust-pipeline cat /proc/mounts | grep models
+   ```
+
+2. **Memory issues**
+   ```bash
+   # Check container memory usage
+   docker stats rust-pipeline
+   
+   # Reduce model context size
+   docker exec rust-pipeline python -c "
+   import os
+   print(f'Context size: {os.environ.get(\"LLM_CONTEXT_SIZE\", \"default\")}')
+   "
+   ```
+
+3. **Crawl4AI browser issues**
+   ```bash
+   # Check browser installation
+   docker exec rust-pipeline /usr/bin/chromium --version
+   docker exec rust-pipeline python -m playwright install --help
+   ```
+
+### Debug Mode
+```bash
+# Run with debug logging
+docker run --rm \
+  -e LOG_LEVEL=DEBUG \
+  -v $(pwd)/output:/app/output \
+  -v ~/models:/app/models:ro \
+  rust-crate-pipeline:1.5.1 \
+  --limit 10 --log-level DEBUG
+```
+
+## Security Considerations
+
+1. **Non-root user**: Container runs as `pipelineuser` (UID 1000)
+2. **Read-only model mount**: Models are mounted read-only
+3. **No user site-packages**: `PYTHONNOUSERSITE=1` prevents loading user packages
+4. **Hash randomization**: `PYTHONHASHSEED=random` for security
+
+## Production Recommendations
+
+1. **Use specific tags**: Pin to `rust-crate-pipeline:1.5.1` instead of `latest`
+2. **Resource monitoring**: Use proper monitoring for CPU/memory usage
+3. **Log rotation**: Configure log rotation for long-running containers
+4. **Health checks**: Monitor container health endpoints
+5. **Security updates**: Regularly update base images
+
+## Version Information
+
+- **Image Version**: 1.5.1
+- **Base Image**: python:3.11.9-slim-bookworm
+- **Python Version**: 3.11.9
+- **Crawl4AI**: Latest compatible version
+- **Model Format**: GGUF (llama-cpp-python compatible)
+
+## Support
+
+For issues or questions:
+- GitHub Issues: https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/issues
+- Documentation: https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/blob/main/README.md

{rust_crate_pipeline-1.4.0/rust_crate_pipeline.egg-info → rust_crate_pipeline-1.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rust-crate-pipeline
-Version: 1.4.0
+Version: 1.5.2
 Summary: A comprehensive system for gathering, enriching, and analyzing metadata for Rust crates using AI-powered insights
 Home-page: https://github.com/Superuser666-Sigil/SigilDERG-Data_Production
 Author: SuperUser666-Sigil
@@ -51,11 +51,13 @@ Dynamic: requires-python
 
 [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyPI Package](https://img.shields.io/badge/PyPI-v1.4.0-green.svg)](https://pypi.org/project/rust-crate-pipeline/)
+[![PyPI Package](https://img.shields.io/badge/PyPI-v1.5.1-green.svg)](https://pypi.org/project/rust-crate-pipeline/)
 [![Docker Ready](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com/)
 [![Rule Zero Compliant](https://img.shields.io/badge/Rule%20Zero-Compliant-gold.svg)](https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/blob/main/SYSTEM_AUDIT_REPORT.md)
 
-A production-ready, Rule Zero-compliant pipeline for comprehensive Rust crate analysis, featuring AI-powered insights, dependency mapping, and automated data enrichment. Designed for researchers, developers, and data scientists studying the Rust ecosystem.
+A production-ready, Rule Zero-compliant pipeline for comprehensive Rust crate analysis, featuring **AI-powered insights**, **enhanced web scraping with Crawl4AI**, dependency mapping, and automated data enrichment. Designed for researchers, developers, and data scientists studying the Rust ecosystem.
+
+**🆕 New in v1.5.1**: Model path standardization, improved GGUF configuration consistency, and enhanced Rule Zero alignment.
 
 📦 **Available on PyPI:** [rust-crate-pipeline](https://pypi.org/project/rust-crate-pipeline/)
 
@@ -126,6 +128,25 @@ python3 -m rust_crate_pipeline --skip-ai --limit 50
 ### 4. Advanced Usage
 
 ```bash
+# Enhanced web scraping with Crawl4AI (default in v1.5.0)
+python3 -m rust_crate_pipeline --enable-crawl4ai --limit 20
+
+# Disable Crawl4AI for basic scraping only
+python3 -m rust_crate_pipeline --disable-crawl4ai --limit 20
+
+# Custom Crawl4AI model configuration
+python3 -m rust_crate_pipeline \
+    --enable-crawl4ai \
+    --crawl4ai-model "~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf" \
+    --limit 10
+
+# Sigil Protocol with enhanced scraping
+python3 -m rust_crate_pipeline \
+    --enable-sigil-protocol \
+    --enable-crawl4ai \
+    --skip-ai \
+    --limit 5
+
 # Custom configuration
 python3 -m rust_crate_pipeline \
     --limit 100 \
@@ -147,7 +168,16 @@ python3 -m rust_crate_pipeline \
 
 ## 🎯 Features
 
-*Available in the latest version: [rust-crate-pipeline v1.4.0](https://pypi.org/project/rust-crate-pipeline/)*
+*Available in the latest version: [rust-crate-pipeline v1.5.1](https://pypi.org/project/rust-crate-pipeline/)*
+
+### 🌐 Enhanced Web Scraping (New in v1.5.0)
+
+- **Crawl4AI Integration**: Advanced web scraping with AI-powered content extraction
+- **JavaScript Rendering**: Playwright-powered browser automation for dynamic content
+- **Smart Content Analysis**: LLM-enhanced README and documentation parsing
+- **Structured Data Extraction**: Intelligent parsing of docs.rs and technical documentation
+- **Quality Scoring**: Automated content quality assessment and validation
+- **Graceful Fallbacks**: Automatic degradation to basic scraping when needed
 
 ### 📊 Data Collection & Analysis
 
@@ -171,8 +201,35 @@ python3 -m rust_crate_pipeline \
 - **Robust error handling**: Graceful degradation and comprehensive logging
 - **Progress checkpointing**: Automatic saving for long-running processes
 - **Docker ready**: Full container support with optimized configurations
+- **Rule Zero Compliance**: Full transparency and audit trail support
 
-## 💻 System Requirements
+## � Recent Updates
+
+### Version 1.5.1 - Configuration Standardization (Latest)
+- 🔧 **Model Path Consistency**: Standardized all configuration to use GGUF model paths (`~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf`)
+- ⚖️ **Rule Zero Alignment**: Enhanced compliance with Rule Zero principles for transparency and validation
+- 📝 **Documentation Updates**: Comprehensive updates to reflect proper model configuration practices
+- 🧪 **Test Standardization**: Updated all test files to use consistent GGUF model paths
+- 🚀 **CLI Consistency**: Ensured all CLI defaults and help text reflect correct model paths
+
+### Version 1.5.0 - Enhanced Web Scraping
+- 🚀 **Crawl4AI Integration**: Advanced web scraping with AI-powered content extraction
+- 🌐 **JavaScript Rendering**: Playwright-powered browser automation for dynamic content
+- 🧠 **LLM-Enhanced Parsing**: AI-powered README and documentation analysis
+- 📊 **Structured Data Extraction**: Intelligent parsing of docs.rs and technical documentation
+- ⚡ **Async Processing**: High-performance concurrent web scraping
+- 🛡️ **Graceful Fallbacks**: Automatic degradation to basic scraping when needed
+
+### Version 1.4.0 - Rule Zero Compliance
+- 🏆 **Rule Zero Certification**: Complete alignment audit and compliance verification
+- 🧪 **100% Test Coverage**: All 22 tests passing with comprehensive validation
+- 🔄 **Thread-Free Architecture**: Pure asyncio implementation for better performance
+- 📦 **PyPI Integration**: Official package availability with easy installation
+- 🐳 **Docker Support**: Full containerization with production-ready configurations
+
+*For complete version history, see [CHANGELOG.md](CHANGELOG.md)*
+
+## �💻 System Requirements
 
 ### Minimum Requirements
 
@@ -193,12 +250,21 @@ python3 -m rust_crate_pipeline \
 Core dependencies are automatically installed:
 
 ```bash
+# Core functionality
 requests>=2.28.0
 requests-cache>=0.9.0
 beautifulsoup4>=4.11.0
 tqdm>=4.64.0
+
+# AI and LLM processing
 llama-cpp-python>=0.2.0
 tiktoken>=0.4.0
+
+# Enhanced web scraping (New in v1.5.0)
+crawl4ai>=0.6.0
+playwright>=1.49.0
+
+# System utilities
 psutil>=5.9.0
 python-dateutil>=2.8.0
 ```
@@ -219,6 +285,11 @@ python-dateutil>=2.8.0
 | `--log-level` | str | INFO | Logging verbosity |
 | `--skip-ai` | flag | False | Skip AI enrichment |
 | `--skip-source-analysis` | flag | False | Skip source code analysis |
+| `--enable-crawl4ai` | flag | True | Enable enhanced web scraping (default) |
+| `--disable-crawl4ai` | flag | False | Disable Crawl4AI, use basic scraping |
+| `--crawl4ai-model` | str | ~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf | GGUF model path for content analysis |
+| `--enable-sigil-protocol` | flag | False | Enable Rule Zero compliance mode |
+| `--sigil-mode` | str | enhanced | Sigil processing mode |
 | `--crate-list` | list | None | Specific crates to process |
 | `--config-file` | str | None | JSON configuration file |
 
@@ -254,7 +325,9 @@ Create a JSON configuration file for custom settings:
     "batch_size": 10,
     "github_min_remaining": 500,
     "cache_ttl": 7200,
-    "model_path": "~/models/your-model.gguf"
+    "model_path": "~/models/your-model.gguf",    "enable_crawl4ai": true,
+    "crawl4ai_model": "~/models/deepseek/deepseek-coder-6.7b-instruct.Q4_K_M.gguf",
+    "crawl4ai_timeout": 30
 }
 ```