rust-crate-pipeline 1.1.1__py3-none-any.whl → 1.2.1__py3-none-any.whl

rust_crate_pipeline-1.2.1.dist-info/METADATA

@@ -0,0 +1,573 @@
+Metadata-Version: 2.4
+Name: rust-crate-pipeline
+Version: 1.2.1
+Summary: A comprehensive system for gathering, enriching, and analyzing metadata for Rust crates using AI-powered insights
+Home-page: https://github.com/DaveTmire85/SigilDERG-Data_Production
+Author: SuperUser666-Sigil
+Author-email: SuperUser666-Sigil <miragemodularframework@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/DaveTmire85/SigilDERG-Data_Production
+Project-URL: Documentation, https://github.com/DaveTmire85/SigilDERG-Data_Production#readme
+Project-URL: Repository, https://github.com/DaveTmire85/SigilDERG-Data_Production
+Project-URL: Bug Tracker, https://github.com/DaveTmire85/SigilDERG-Data_Production/issues
+Keywords: rust,crates,metadata,ai,analysis,pipeline,dependencies
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Software Development :: Build Tools
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28.0
+Requires-Dist: requests-cache>=1.0.0
+Requires-Dist: beautifulsoup4>=4.11.0
+Requires-Dist: tqdm>=4.64.0
+Requires-Dist: llama-cpp-python>=0.2.0
+Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: python-dateutil>=2.8.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: black>=22.0.0; extra == "dev"
+Requires-Dist: isort>=5.10.0; extra == "dev"
+Provides-Extra: advanced
+Requires-Dist: radon>=6.0.0; extra == "advanced"
+Requires-Dist: rustworkx>=0.13.0; extra == "advanced"
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# Rust Crate Pipeline
+
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI Ready](https://img.shields.io/badge/PyPI-Ready-green.svg)](https://pypi.org/)
+[![Docker Ready](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com/)
+
+A production-ready pipeline for comprehensive Rust crate analysis, featuring AI-powered insights, dependency mapping, and automated data enrichment. Designed for researchers, developers, and data scientists studying the Rust ecosystem.
+
+## 🚀 Quick Start
+
+### 1. Installation
+
+#### From PyPI (Recommended)
+```bash
+pip install rust-crate-pipeline
+```
+
+#### From Source
+```bash
+git clone https://github.com/DaveTmire85/SigilDERG-Data_Production.git
+cd SigilDERG-Data_Production
+pip install -e .
+```
+
+#### Development Installation
+```bash
+git clone https://github.com/DaveTmire85/SigilDERG-Data_Production.git
+cd SigilDERG-Data_Production
+pip install -e ".[dev]"
+```
+
+### 2. GitHub Token Setup
+
+The pipeline requires a GitHub Personal Access Token for optimal performance:
+
+```bash
+# Interactive setup (Linux/Unix)
+chmod +x setup_github_token.sh
+./setup_github_token.sh
+
+# Manual setup
+export GITHUB_TOKEN="your_token_here"
+echo 'export GITHUB_TOKEN="your_token_here"' >> ~/.bashrc
+
+# Verify setup
+python3 check_github_token.py
+```
+
+**Get your token at**: [GitHub Settings](https://github.com/settings/tokens)  
+**Required scopes**: `public_repo`, `read:user`
+
+### 3. Basic Usage
+
+```bash
+# Standard mode
+python3 -m rust_crate_pipeline
+
+# Production mode (reduced warnings, optimized settings)
+python3 run_production.py
+
+# Process only 20 crates for testing
+python3 -m rust_crate_pipeline --limit 20
+
+# Skip AI processing for faster metadata-only collection
+python3 -m rust_crate_pipeline --skip-ai --limit 50
+```
+
+### 4. Advanced Usage
+
+```bash
+# Custom configuration
+python3 -m rust_crate_pipeline \
+    --limit 100 \
+    --batch-size 5 \
+    --workers 2 \
+    --log-level DEBUG \
+    --output-dir ./results
+
+# Process specific crates
+python3 -m rust_crate_pipeline \
+    --crate-list serde tokio actix-web reqwest \
+    --output-dir ./specific_crates
+
+# Use custom model and config
+python3 -m rust_crate_pipeline \
+    --model-path ./my-model.gguf \
+    --config-file ./custom_config.json
+```
+
+## 🎯 Features
+
+### 📊 Data Collection & Analysis
+
+- **Multi-source metadata**: crates.io, GitHub, lib.rs integration
+- **Dependency mapping**: Complete dependency graphs and analysis
+- **Code extraction**: Automatic Rust code example extraction
+- **Security scanning**: Vulnerability and security pattern analysis
+- **Performance metrics**: Lines of code, complexity, API surface analysis
+
+### 🤖 AI-Powered Enrichment
+
+- **Smart categorization**: Automatic crate classification (Web, ML, Database, etc.)
+- **Feature summarization**: AI-generated explanations and insights
+- **Content optimization**: Intelligent README section preservation
+- **Factual pairs**: Training data generation for fact verification
+
+### ⚡ Production Features
+
+- **Automatic GitHub token detection**: Seamless setup and validation
+- **Smart rate limiting**: Respects GitHub API limits with intelligent backoff
+- **Robust error handling**: Graceful degradation and comprehensive logging
+- **Progress checkpointing**: Automatic saving for long-running processes
+- **Docker ready**: Full container support with optimized configurations
+
+## 💻 System Requirements
+
+### Minimum Requirements
+
+- **Python**: 3.8+
+- **Memory**: 4GB RAM
+- **Storage**: 2GB free space
+- **Network**: Stable internet connection
+
+### Recommended Setup
+
+- **Python**: 3.10+
+- **Memory**: 8GB+ RAM
+- **Storage**: 10GB+ free space (SSD preferred)
+- **GitHub Token**: For enhanced API access (5000 vs 60 requests/hour)
+
+### Dependencies
+
+Core dependencies are automatically installed:
+
+```bash
+requests>=2.28.0
+requests-cache>=0.9.0
+beautifulsoup4>=4.11.0
+tqdm>=4.64.0
+llama-cpp-python>=0.2.0
+tiktoken>=0.4.0
+psutil>=5.9.0
+python-dateutil>=2.8.0
+```
+
+## ⚙️ Configuration & Usage
+
+### Command Line Options
+
+| Argument | Type | Default | Description |
+|----------|------|---------|-------------|
+| `--limit` | int | None | Limit number of crates to process |
+| `--batch-size` | int | 10 | Crates processed per batch |
+| `--workers` | int | 4 | Parallel workers for API requests |
+| `--output-dir` | str | auto | Custom output directory |
+| `--model-path` | str | default | Path to LLM model file |
+| `--max-tokens` | int | 256 | Maximum tokens for LLM generation |
+| `--checkpoint-interval` | int | 10 | Save progress every N crates |
+| `--log-level` | str | INFO | Logging verbosity |
+| `--skip-ai` | flag | False | Skip AI enrichment |
+| `--skip-source-analysis` | flag | False | Skip source code analysis |
+| `--crate-list` | list | None | Specific crates to process |
+| `--config-file` | str | None | JSON configuration file |
+
+### Production Mode
+
+Production mode provides optimized settings with reduced warnings:
+
+```bash
+# Using production launcher
+python3 run_production.py [OPTIONS]
+
+# Using environment variable
+PRODUCTION=true python3 -m rust_crate_pipeline
+
+# Docker production mode
+docker run -e PRODUCTION=true -e GITHUB_TOKEN="token" your-image
+```
+
+**Production optimizations:**
+
+- Reduced retry attempts (3→2) to minimize warnings
+- Smart GitHub API rate limiting with proactive pausing
+- Enhanced logging with appropriate levels
+- Optimized timeout and backoff strategies
+
+### Configuration Files
+
+Create a JSON configuration file for custom settings:
+
+```json
+{
+    "max_retries": 2,
+    "batch_size": 10,
+    "github_min_remaining": 500,
+    "cache_ttl": 7200,
+    "model_path": "~/models/your-model.gguf"
+}
+```
+
+Use with: `python3 -m rust_crate_pipeline --config-file config.json`
+
+## 🐳 Docker Deployment
+
+### Using Docker Compose (Recommended)
+
+```bash
+# Set up environment
+echo "GITHUB_TOKEN=your_token_here" > .env
+
+# Run with compose
+docker-compose up -d
+
+# Monitor logs
+docker-compose logs -f
+```
+
+### Manual Docker Commands
+
+```bash
+# Build image
+docker build -t rust-crate-pipeline .
+
+# Run container
+docker run -e GITHUB_TOKEN="your_token" \
+           -e PRODUCTION=true \
+           -v $(pwd)/output:/app/output \
+           rust-crate-pipeline
+
+# Background execution
+docker run -d --name pipeline \
+           -e GITHUB_TOKEN="your_token" \
+           rust-crate-pipeline
+```
+
+### Docker Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `GITHUB_TOKEN` | GitHub Personal Access Token | Required |
+| `PRODUCTION` | Enable production mode | `false` |
+| `PYTHONUNBUFFERED` | Force unbuffered output | `1` |
+
+## 📊 Output & Data Format
+
+### Output Structure
+
+```
+output/
+├── enriched_crates_YYYYMMDD_HHMMSS.json    # Main results
+├── metadata_YYYYMMDD_HHMMSS.json           # Raw metadata
+├── errors_YYYYMMDD_HHMMSS.log              # Error log
+└── checkpoints/
+    └── checkpoint_N.json                    # Progress saves
+```
+
+### Data Schema
+
+Each processed crate includes:
+
+```json
+{
+    "name": "serde",
+    "version": "1.0.193",
+    "description": "A generic serialization/deserialization framework",
+    "repository": "https://github.com/serde-rs/serde",
+    "downloads": 50000000,
+    "github_stars": 8500,
+    "category": "Serialization",
+    "use_case": "Data serialization and deserialization",
+    "feature_summary": "Compile-time serialization framework...",
+    "dependencies": [...],
+    "security_analysis": {...},
+    "source_metrics": {...}
+}
+```
+
+## 🔍 Monitoring & Troubleshooting
+
+### Common Issues & Solutions
+
+#### GitHub Token Problems
+
+```bash
+# Check token status
+python3 check_github_token.py
+
+# Common error: Rate limit warnings
+[WARNING] GitHub API rate limit low: 60 remaining
+# Solution: Set GITHUB_TOKEN environment variable
+
+# Common error: Invalid token
+[ERROR] GitHub token is invalid or expired
+# Solution: Generate new token at https://github.com/settings/tokens
+```
+
+#### LLM Validation Retries
+
+```bash
+# Common warning: Validation failures
+[WARNING] Validation failed on attempt 1/3. Retrying...
+# Solution: Use production mode to reduce retry warnings
+PRODUCTION=true python3 -m rust_crate_pipeline
+```
+
+#### Resource Issues
+
+```bash
+# Memory usage optimization
+python3 -m rust_crate_pipeline --batch-size 3
+
+# Disk space monitoring
+df -h .  # Check available space
+
+# Network timeout handling
+python3 -m rust_crate_pipeline --log-level DEBUG
+```
+
+### Performance Monitoring
+
+#### Processing Times (Typical)
+
+- **Metadata only**: 2-3 seconds per crate
+- **With AI enrichment**: 15-30 seconds per crate  
+- **Full analysis**: 45-60 seconds per crate
+
+#### Resource Usage
+
+- **Memory**: 2-4GB during processing
+- **Storage**: 10-50MB per crate (temporary files)
+- **Network**: 1-5MB per crate (API calls)
+
+#### Monitoring Commands
+
+```bash
+# Check process status
+ps aux | grep rust_crate_pipeline
+
+# Monitor resource usage
+top -p $(pgrep -f rust_crate_pipeline)
+
+# Check logs
+tail -f pipeline.log
+
+# Docker monitoring
+docker stats pipeline
+```
+
+## 🚀 Deployment Guide
+
+### SSH/Remote Server Deployment
+
+```bash
+# Background execution with logging
+nohup python3 run_production.py > pipeline.log 2>&1 &
+
+# Monitor progress
+tail -f pipeline.log
+
+# Check process
+jobs
+ps aux | grep rust_crate_pipeline
+```
+
+### Systemd Service (Linux)
+
+Create `/etc/systemd/system/rust-crate-pipeline.service`:
+
+```ini
+[Unit]
+Description=Rust Crate Data Pipeline
+After=network.target
+
+[Service]
+Type=simple
+User=your-username
+WorkingDirectory=/path/to/pipeline
+Environment=GITHUB_TOKEN=your_token_here
+Environment=PRODUCTION=true
+ExecStart=/usr/bin/python3 run_production.py
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Enable and start:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable rust-crate-pipeline
+sudo systemctl start rust-crate-pipeline
+sudo systemctl status rust-crate-pipeline
+```
+
+## 🏗️ Architecture
+
+### Core Components
+
+1. **CrateDataPipeline**: Main orchestration class that coordinates all processing
+2. **LLMEnricher**: Handles AI-powered enrichment using local LLM models
+3. **CrateAPIClient**: Manages API interactions with crates.io and fallback sources
+4. **GitHubBatchClient**: Optimized GitHub API client with rate limiting
+5. **SourceAnalyzer**: Analyzes source code metrics and complexity
+6. **SecurityAnalyzer**: Checks for security vulnerabilities and patterns
+7. **UserBehaviorAnalyzer**: Tracks community engagement and version adoption
+8. **DependencyAnalyzer**: Builds and analyzes dependency relationships
+
+### Processing Flow
+
+```
+1. Crate Discovery → 2. Metadata Fetching → 3. AI Enrichment
+        ↓                      ↓                    ↓
+4. Source Analysis → 5. Security Scanning → 6. Community Analysis
+        ↓                      ↓                    ↓
+7. Dependency Mapping → 8. Data Aggregation → 9. Report Generation
+```
+
+### Project Structure
+
+```
+rust_crate_pipeline/
+├── __init__.py              # Package initialization
+├── __main__.py              # Entry point for python -m execution
+├── main.py                  # CLI interface and main execution logic
+├── config.py                # Configuration classes and data models
+├── pipeline.py              # Main orchestration and workflow management
+├── ai_processing.py         # LLM integration and AI-powered enrichment
+├── network.py               # API clients and HTTP request handling
+├── analysis.py              # Source code, security, and dependency analysis
+├── github_token_checker.py  # Token validation and setup
+├── production_config.py     # Production optimizations
+└── utils/                   # Utility functions
+    ├── logging_utils.py     # Logging configuration and decorators
+    └── file_utils.py        # File operations and disk management
+```
+
+## 🧪 API Usage
+
+### Programmatic Usage
+
+```python
+from rust_crate_pipeline import CrateDataPipeline, PipelineConfig
+
+# Create custom configuration
+config = PipelineConfig(
+    batch_size=5,
+    max_tokens=512,
+    model_path="/path/to/model.gguf"
+)
+
+# Initialize and run pipeline
+pipeline = CrateDataPipeline(config)
+pipeline.run()
+
+# Or use individual components
+from rust_crate_pipeline import LLMEnricher, SourceAnalyzer
+
+enricher = LLMEnricher(config)
+analyzer = SourceAnalyzer()
+```
+
+### Custom Processing
+
+```python
+# Process specific crates with custom options
+pipeline = CrateDataPipeline(
+    config,
+    limit=50,
+    crate_list=["serde", "tokio", "actix-web"],
+    skip_ai=False,
+    output_dir="./custom_analysis"
+)
+```
+
+## 🔧 Development & Contributing
+
+### Development Setup
+
+```bash
+# Clone and install
+git clone https://github.com/DaveTmire85/SigilDERG-Data_Production.git
+cd SigilDERG-Data_Production
+pip install -r requirements.txt
+
+# Run tests
+python3 test_optimizations.py
+python3 test_token_integration.py
+
+# Verify installation
+python3 check_github_token.py
+```
+
+### Adding Features
+
+1. Implement new analyzer in `analysis.py`
+2. Add configuration options to `config.py`
+3. Integrate with pipeline in `pipeline.py`
+4. Add CLI arguments in `main.py`
+5. Update tests and documentation
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## 🙏 Acknowledgments
+
+- **Rust Community** for the excellent crates ecosystem
+- **crates.io** for comprehensive API access
+- **GitHub** for repository metadata and community data
+- **Deepseek** for powerful code-focused language models
+- **llama.cpp** team for efficient local inference
+
+## 📞 Support
+
+- **Issues**: Report bugs and request features
+- **Documentation**: Complete guides and API reference
+- **Community**: Join discussions and get help
+
+---
+
+**Ready to analyze the Rust ecosystem! 🦀✨**

{rust_crate_pipeline-1.1.1.dist-info → rust_crate_pipeline-1.2.1.dist-info}/RECORD

@@ -1,17 +1,19 @@
 rust_crate_pipeline/__init__.py,sha256=m9fb1WGbyOimxK2e18FSgvLWGYBwbLoHM_mscr-nAPs,1429
 rust_crate_pipeline/__main__.py,sha256=fYgtPofuk4vkwiZ7ELP4GVMNj_QiKmZMSlvhzsNGuDs,155
-rust_crate_pipeline/ai_processing.py,sha256=kmV6RfhwRVYQyp55Eez8R0jNAVJB8sdfIB1-60j1Eio,18027
+rust_crate_pipeline/ai_processing.py,sha256=zs0eHxnoSajRqpUoZ5fXLxrK68jZ-J1Vavtsujskc9Q,18821
 rust_crate_pipeline/analysis.py,sha256=ijP4zp3cFnN09nZkeCluyAvbyAtAW_M2YSxALpQX8LY,18615
 rust_crate_pipeline/config.py,sha256=r4Y_5SD-lfrM1112edk9T0S0MiVxaNSSHk4q2yDrM88,1528
-rust_crate_pipeline/main.py,sha256=vPLvuwekNpwSUHEAc6lnzQ0q3QU-YZS9ZIijhkkfRzI,5384
+rust_crate_pipeline/github_token_checker.py,sha256=MJqHP8J84NEZ6nzdutpC7iRnsP0kyqscjLUosvmI4MI,3768
+rust_crate_pipeline/main.py,sha256=J8ORQA6s3wyWw2R3oB_IEm2J5tx1CFdspw5kb5Ep8zQ,6323
 rust_crate_pipeline/network.py,sha256=t_G8eh_WHNugm_laMftcWVbHsmP0bOlTPnVW9DqF6SU,13375
 rust_crate_pipeline/pipeline.py,sha256=Uwfw4uLL3aN1gJl5xSwvvyaY9ceeP7LVr02IzNx0tPM,12033
-rust_crate_pipeline/version.py,sha256=np_eAssJT7w0mfKLkvPGj07FJndSt79OH1PjHEaXb_s,542
+rust_crate_pipeline/production_config.py,sha256=2GT8bxytcrMRrcfjzpay5RTtATE3rbmDvNUBvVhrYSQ,2472
+rust_crate_pipeline/version.py,sha256=8LymcBLNSPWijVTHpd9sS_Aqp8vRoMyZLCupAOnQkfs,1022
 rust_crate_pipeline/utils/file_utils.py,sha256=lnHeLrt1JYaQhRDKtA1TWR2HIyRO8zwOyWb-KmAmWgk,2126
 rust_crate_pipeline/utils/logging_utils.py,sha256=O4Jnr_k9dBchrVqXf-vqtDKgizDtL_ljh8g7G2VCX_c,2241
-rust_crate_pipeline-1.1.1.dist-info/licenses/LICENSE,sha256=tpd4XNpbssrSx9-iErATOLrOh0ivNPfO2I5MAPUpats,1088
-rust_crate_pipeline-1.1.1.dist-info/METADATA,sha256=96NawdhQK_JtrzH7uQYVxD4JJtsr-p8ask9tf6QmgTc,15260
-rust_crate_pipeline-1.1.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-rust_crate_pipeline-1.1.1.dist-info/entry_points.txt,sha256=9Rr_IRuFRIridXxUSdEJbB3ba0NnpEfKmknZXFdYRC0,70
-rust_crate_pipeline-1.1.1.dist-info/top_level.txt,sha256=GUdB7RyxHLhijQxui_KTy3B8p_L2APui9C6RYa0FuaE,20
-rust_crate_pipeline-1.1.1.dist-info/RECORD,,
+rust_crate_pipeline-1.2.1.dist-info/licenses/LICENSE,sha256=tpd4XNpbssrSx9-iErATOLrOh0ivNPfO2I5MAPUpats,1088
+rust_crate_pipeline-1.2.1.dist-info/METADATA,sha256=zRFlOck4m9GDEHrmBM_cvzKVVKD-zBK4hzfClVbhuq0,16741
+rust_crate_pipeline-1.2.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+rust_crate_pipeline-1.2.1.dist-info/entry_points.txt,sha256=9Rr_IRuFRIridXxUSdEJbB3ba0NnpEfKmknZXFdYRC0,70
+rust_crate_pipeline-1.2.1.dist-info/top_level.txt,sha256=GUdB7RyxHLhijQxui_KTy3B8p_L2APui9C6RYa0FuaE,20
+rust_crate_pipeline-1.2.1.dist-info/RECORD,,