rust-crate-pipeline 1.4.0__py3-none-any.whl → 1.4.1__py3-none-any.whl

rust_crate_pipeline-1.4.0.dist-info/METADATA

@@ -1,585 +0,0 @@
-Metadata-Version: 2.4
-Name: rust-crate-pipeline
-Version: 1.4.0
-Summary: A comprehensive system for gathering, enriching, and analyzing metadata for Rust crates using AI-powered insights
-Home-page: https://github.com/Superuser666-Sigil/SigilDERG-Data_Production
-Author: SuperUser666-Sigil
-Author-email: SuperUser666-Sigil <miragemodularframework@gmail.com>
-License-Expression: MIT
-Project-URL: Homepage, https://github.com/Superuser666-Sigil/SigilDERG-Data_Production
-Project-URL: Documentation, https://github.com/Superuser666-Sigil/SigilDERG-Data_Production#readme
-Project-URL: Repository, https://github.com/Superuser666-Sigil/SigilDERG-Data_Production
-Project-URL: Bug Tracker, https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/issues
-Keywords: rust,crates,metadata,ai,analysis,pipeline,dependencies
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Developers
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Classifier: Topic :: Software Development :: Build Tools
-Classifier: Topic :: Software Development :: Quality Assurance
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: requests>=2.28.0
-Requires-Dist: requests-cache>=1.0.0
-Requires-Dist: beautifulsoup4>=4.11.0
-Requires-Dist: tqdm>=4.64.0
-Requires-Dist: llama-cpp-python>=0.2.0
-Requires-Dist: tiktoken>=0.5.0
-Requires-Dist: psutil>=5.9.0
-Requires-Dist: python-dateutil>=2.8.0
-Provides-Extra: dev
-Requires-Dist: pytest>=7.0.0; extra == "dev"
-Requires-Dist: black>=22.0.0; extra == "dev"
-Requires-Dist: isort>=5.10.0; extra == "dev"
-Provides-Extra: advanced
-Requires-Dist: radon>=6.0.0; extra == "advanced"
-Requires-Dist: rustworkx>=0.13.0; extra == "advanced"
-Dynamic: author
-Dynamic: home-page
-Dynamic: license-file
-Dynamic: requires-python
-
-# Rust Crate Pipeline
-
-[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyPI Package](https://img.shields.io/badge/PyPI-v1.4.0-green.svg)](https://pypi.org/project/rust-crate-pipeline/)
-[![Docker Ready](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://docker.com/)
-[![Rule Zero Compliant](https://img.shields.io/badge/Rule%20Zero-Compliant-gold.svg)](https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/blob/main/SYSTEM_AUDIT_REPORT.md)
-
-A production-ready, Rule Zero-compliant pipeline for comprehensive Rust crate analysis, featuring AI-powered insights, dependency mapping, and automated data enrichment. Designed for researchers, developers, and data scientists studying the Rust ecosystem.
-
-📦 **Available on PyPI:** [rust-crate-pipeline](https://pypi.org/project/rust-crate-pipeline/)
-
-## 🚀 Quick Start
-
-### 1. Installation
-
-#### From PyPI (Recommended)
-
-```bash
-pip install rust-crate-pipeline
-```
-
-For the latest version, visit: [rust-crate-pipeline on PyPI](https://pypi.org/project/rust-crate-pipeline/)
-
-#### From Source
-
-```bash
-git clone https://github.com/Superuser666-Sigil/SigilDERG-Data_Production.git
-cd SigilDERG-Data_Production
-pip install -e .
-```
-
-#### Development Installation
-
-```bash
-git clone https://github.com/Superuser666-Sigil/SigilDERG-Data_Production.git
-cd SigilDERG-Data_Production
-pip install -e ".[dev]"
-```
-
-### 2. GitHub Token Setup
-
-The pipeline requires a GitHub Personal Access Token for optimal performance:
-
-```bash
-# Interactive setup (Linux/Unix)
-chmod +x setup_github_token.sh
-./setup_github_token.sh
-
-# Manual setup
-export GITHUB_TOKEN="your_token_here"
-echo 'export GITHUB_TOKEN="your_token_here"' >> ~/.bashrc
-
-# Verify setup
-python3 check_github_token.py
-```
-
-**Get your token at**: [GitHub Settings](https://github.com/settings/tokens)  
-**Required scopes**: `public_repo`, `read:user`
-
-### 3. Basic Usage
-
-```bash
-# Standard mode
-python3 -m rust_crate_pipeline
-
-# Production mode (reduced warnings, optimized settings)
-python3 run_production.py
-
-# Process only 20 crates for testing
-python3 -m rust_crate_pipeline --limit 20
-
-# Skip AI processing for faster metadata-only collection
-python3 -m rust_crate_pipeline --skip-ai --limit 50
-```
-
-### 4. Advanced Usage
-
-```bash
-# Custom configuration
-python3 -m rust_crate_pipeline \
-    --limit 100 \
-    --batch-size 5 \
-    --workers 2 \
-    --log-level DEBUG \
-    --output-dir ./results
-
-# Process specific crates
-python3 -m rust_crate_pipeline \
-    --crate-list serde tokio actix-web reqwest \
-    --output-dir ./specific_crates
-
-# Use custom model and config
-python3 -m rust_crate_pipeline \
-    --model-path ./my-model.gguf \
-    --config-file ./custom_config.json
-```
-
-## 🎯 Features
-
-*Available in the latest version: [rust-crate-pipeline v1.4.0](https://pypi.org/project/rust-crate-pipeline/)*
-
-### 📊 Data Collection & Analysis
-
-- **Multi-source metadata**: crates.io, GitHub, lib.rs integration
-- **Dependency mapping**: Complete dependency graphs and analysis
-- **Code extraction**: Automatic Rust code example extraction
-- **Security scanning**: Vulnerability and security pattern analysis
-- **Performance metrics**: Lines of code, complexity, API surface analysis
-
-### 🤖 AI-Powered Enrichment
-
-- **Smart categorization**: Automatic crate classification (Web, ML, Database, etc.)
-- **Feature summarization**: AI-generated explanations and insights
-- **Content optimization**: Intelligent README section preservation
-- **Factual pairs**: Training data generation for fact verification
-
-### ⚡ Production Features
-
-- **Automatic GitHub token detection**: Seamless setup and validation
-- **Smart rate limiting**: Respects GitHub API limits with intelligent backoff
-- **Robust error handling**: Graceful degradation and comprehensive logging
-- **Progress checkpointing**: Automatic saving for long-running processes
-- **Docker ready**: Full container support with optimized configurations
-
-## 💻 System Requirements
-
-### Minimum Requirements
-
-- **Python**: 3.8+
-- **Memory**: 4GB RAM
-- **Storage**: 2GB free space
-- **Network**: Stable internet connection
-
-### Recommended Setup
-
-- **Python**: 3.10+
-- **Memory**: 8GB+ RAM
-- **Storage**: 10GB+ free space (SSD preferred)
-- **GitHub Token**: For enhanced API access (5000 vs 60 requests/hour)
-
-### Dependencies
-
-Core dependencies are automatically installed:
-
-```bash
-requests>=2.28.0
-requests-cache>=0.9.0
-beautifulsoup4>=4.11.0
-tqdm>=4.64.0
-llama-cpp-python>=0.2.0
-tiktoken>=0.4.0
-psutil>=5.9.0
-python-dateutil>=2.8.0
-```
-
-## ⚙️ Configuration & Usage
-
-### Command Line Options
-
-| Argument | Type | Default | Description |
-|----------|------|---------|-------------|
-| `--limit` | int | None | Limit number of crates to process |
-| `--batch-size` | int | 10 | Crates processed per batch |
-| `--workers` | int | 4 | Parallel workers for API requests |
-| `--output-dir` | str | auto | Custom output directory |
-| `--model-path` | str | default | Path to LLM model file |
-| `--max-tokens` | int | 256 | Maximum tokens for LLM generation |
-| `--checkpoint-interval` | int | 10 | Save progress every N crates |
-| `--log-level` | str | INFO | Logging verbosity |
-| `--skip-ai` | flag | False | Skip AI enrichment |
-| `--skip-source-analysis` | flag | False | Skip source code analysis |
-| `--crate-list` | list | None | Specific crates to process |
-| `--config-file` | str | None | JSON configuration file |
-
-### Production Mode
-
-Production mode provides optimized settings with reduced warnings:
-
-```bash
-# Using production launcher
-python3 run_production.py [OPTIONS]
-
-# Using environment variable
-PRODUCTION=true python3 -m rust_crate_pipeline
-
-# Docker production mode
-docker run -e PRODUCTION=true -e GITHUB_TOKEN="token" your-image
-```
-
-**Production optimizations:**
-
-- Reduced retry attempts (3→2) to minimize warnings
-- Smart GitHub API rate limiting with proactive pausing
-- Enhanced logging with appropriate levels
-- Optimized timeout and backoff strategies
-
-### Configuration Files
-
-Create a JSON configuration file for custom settings:
-
-```json
-{
-    "max_retries": 2,
-    "batch_size": 10,
-    "github_min_remaining": 500,
-    "cache_ttl": 7200,
-    "model_path": "~/models/your-model.gguf"
-}
-```
-
-Use with: `python3 -m rust_crate_pipeline --config-file config.json`
-
-## 🐳 Docker Deployment
-
-### Using Docker Compose (Recommended)
-
-```bash
-# Set up environment
-echo "GITHUB_TOKEN=your_token_here" > .env
-
-# Run with compose
-docker-compose up -d
-
-# Monitor logs
-docker-compose logs -f
-```
-
-### Manual Docker Commands
-
-```bash
-# Build image
-docker build -t rust-crate-pipeline .
-
-# Run container
-docker run -e GITHUB_TOKEN="your_token" \
-           -e PRODUCTION=true \
-           -v $(pwd)/output:/app/output \
-           rust-crate-pipeline
-
-# Background execution
-docker run -d --name pipeline \
-           -e GITHUB_TOKEN="your_token" \
-           rust-crate-pipeline
-```
-
-### Docker Environment Variables
-
-| Variable | Description | Default |
-|----------|-------------|---------|
-| `GITHUB_TOKEN` | GitHub Personal Access Token | Required |
-| `PRODUCTION` | Enable production mode | `false` |
-| `PYTHONUNBUFFERED` | Force unbuffered output | `1` |
-
-## 📊 Output & Data Format
-
-### Output Structure
-
-```text
-output/
-├── enriched_crates_YYYYMMDD_HHMMSS.json    # Main results
-├── metadata_YYYYMMDD_HHMMSS.json           # Raw metadata
-├── errors_YYYYMMDD_HHMMSS.log              # Error log
-└── checkpoints/
-    └── checkpoint_N.json                    # Progress saves
-```
-
-### Data Schema
-
-Each processed crate includes:
-
-```json
-{
-    "name": "serde",
-    "version": "1.0.193",
-    "description": "A generic serialization/deserialization framework",
-    "repository": "https://github.com/serde-rs/serde",
-    "downloads": 50000000,
-    "github_stars": 8500,
-    "category": "Serialization",
-    "use_case": "Data serialization and deserialization",
-    "feature_summary": "Compile-time serialization framework...",
-    "dependencies": [...],
-    "security_analysis": {...},
-    "source_metrics": {...}
-}
-```
-
-## 🔍 Monitoring & Troubleshooting
-
-### Common Issues & Solutions
-
-#### GitHub Token Problems
-
-```bash
-# Check token status
-python3 check_github_token.py
-
-# Common error: Rate limit warnings
-[WARNING] GitHub API rate limit low: 60 remaining
-# Solution: Set GITHUB_TOKEN environment variable
-
-# Common error: Invalid token
-[ERROR] GitHub token is invalid or expired
-# Solution: Generate new token at https://github.com/settings/tokens
-```
-
-#### LLM Validation Retries
-
-```bash
-# Common warning: Validation failures
-[WARNING] Validation failed on attempt 1/3. Retrying...
-# Solution: Use production mode to reduce retry warnings
-PRODUCTION=true python3 -m rust_crate_pipeline
-```
-
-#### Resource Issues
-
-```bash
-# Memory usage optimization
-python3 -m rust_crate_pipeline --batch-size 3
-
-# Disk space monitoring
-df -h .  # Check available space
-
-# Network timeout handling
-python3 -m rust_crate_pipeline --log-level DEBUG
-```
-
-### Performance Monitoring
-
-#### Processing Times (Typical)
-
-- **Metadata only**: 2-3 seconds per crate
-- **With AI enrichment**: 15-30 seconds per crate  
-- **Full analysis**: 45-60 seconds per crate
-
-#### Resource Usage
-
-- **Memory**: 2-4GB during processing
-- **Storage**: 10-50MB per crate (temporary files)
-- **Network**: 1-5MB per crate (API calls)
-
-#### Monitoring Commands
-
-```bash
-# Check process status
-ps aux | grep rust_crate_pipeline
-
-# Monitor resource usage
-top -p $(pgrep -f rust_crate_pipeline)
-
-# Check logs
-tail -f pipeline.log
-
-# Docker monitoring
-docker stats pipeline
-```
-
-## 🚀 Deployment Guide
-
-### SSH/Remote Server Deployment
-
-```bash
-# Background execution with logging
-nohup python3 run_production.py > pipeline.log 2>&1 &
-
-# Monitor progress
-tail -f pipeline.log
-
-# Check process
-jobs
-ps aux | grep rust_crate_pipeline
-```
-
-### Systemd Service (Linux)
-
-Create `/etc/systemd/system/rust-crate-pipeline.service`:
-
-```ini
-[Unit]
-Description=Rust Crate Data Pipeline
-After=network.target
-
-[Service]
-Type=simple
-User=your-username
-WorkingDirectory=/path/to/pipeline
-Environment=GITHUB_TOKEN=your_token_here
-Environment=PRODUCTION=true
-ExecStart=/usr/bin/python3 run_production.py
-Restart=on-failure
-RestartSec=30
-
-[Install]
-WantedBy=multi-user.target
-```
-
-Enable and start:
-
-```bash
-sudo systemctl daemon-reload
-sudo systemctl enable rust-crate-pipeline
-sudo systemctl start rust-crate-pipeline
-sudo systemctl status rust-crate-pipeline
-```
-
-## 🏗️ Architecture
-
-### Core Components
-
-1. **CrateDataPipeline**: Main orchestration class that coordinates all processing
-2. **LLMEnricher**: Handles AI-powered enrichment using local LLM models
-3. **CrateAPIClient**: Manages API interactions with crates.io and fallback sources
-4. **GitHubBatchClient**: Optimized GitHub API client with rate limiting
-5. **SourceAnalyzer**: Analyzes source code metrics and complexity
-6. **SecurityAnalyzer**: Checks for security vulnerabilities and patterns
-7. **UserBehaviorAnalyzer**: Tracks community engagement and version adoption
-8. **DependencyAnalyzer**: Builds and analyzes dependency relationships
-
-### Processing Flow
-
-```text
-1. Crate Discovery → 2. Metadata Fetching → 3. AI Enrichment
-        ↓                      ↓                    ↓
-4. Source Analysis → 5. Security Scanning → 6. Community Analysis
-        ↓                      ↓                    ↓
-7. Dependency Mapping → 8. Data Aggregation → 9. Report Generation
-```
-
-### Project Structure
-
-```text
-rust_crate_pipeline/
-├── __init__.py              # Package initialization
-├── __main__.py              # Entry point for python -m execution
-├── main.py                  # CLI interface and main execution logic
-├── config.py                # Configuration classes and data models
-├── pipeline.py              # Main orchestration and workflow management
-├── ai_processing.py         # LLM integration and AI-powered enrichment
-├── network.py               # API clients and HTTP request handling
-├── analysis.py              # Source code, security, and dependency analysis
-├── github_token_checker.py  # Token validation and setup
-├── production_config.py     # Production optimizations
-└── utils/                   # Utility functions
-    ├── logging_utils.py     # Logging configuration and decorators
-    └── file_utils.py        # File operations and disk management
-```
-
-## 🧪 API Usage
-
-### Programmatic Usage
-
-```python
-from rust_crate_pipeline import CrateDataPipeline, PipelineConfig
-
-# Create custom configuration
-config = PipelineConfig(
-    batch_size=5,
-    max_tokens=512,
-    model_path="/path/to/model.gguf"
-)
-
-# Initialize and run pipeline
-pipeline = CrateDataPipeline(config)
-pipeline.run()
-
-# Or use individual components
-from rust_crate_pipeline import LLMEnricher, SourceAnalyzer
-
-enricher = LLMEnricher(config)
-analyzer = SourceAnalyzer()
-```
-
-### Custom Processing
-
-```python
-# Process specific crates with custom options
-pipeline = CrateDataPipeline(
-    config,
-    limit=50,
-    crate_list=["serde", "tokio", "actix-web"],
-    skip_ai=False,
-    output_dir="./custom_analysis"
-)
-```
-
-## 🔧 Development & Contributing
-
-### Development Setup
-
-```bash
-# Clone and install
-git clone https://github.com/Superuser666-Sigil/SigilDERG-Data_Production.git
-cd SigilDERG-Data_Production
-pip install -r requirements.txt
-
-# Run tests
-python3 test_optimizations.py
-python3 test_token_integration.py
-
-# Verify installation
-python3 check_github_token.py
-```
-
-### Adding Features
-
-1. Implement new analyzer in `analysis.py`
-2. Add configuration options to `config.py`
-3. Integrate with pipeline in `pipeline.py`
-4. Add CLI arguments in `main.py`
-5. Update tests and documentation
-
-## 📄 License
-
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-## 🙏 Acknowledgments
-
-- **Rust Community** for the excellent crates ecosystem
-- **crates.io** for comprehensive API access
-- **GitHub** for repository metadata and community data
-- **Deepseek** for powerful code-focused language models
-- **llama.cpp** team for efficient local inference
-
-## 📞 Support
-
-- **Issues**: Report bugs and request features
-- **Documentation**: Complete guides and API reference
-- **Community**: Join discussions and get help
-
----
-
-## Ready to analyze the Rust ecosystem! 🦀✨
-
-📦 **Get started today:** [Install from PyPI](https://pypi.org/project/rust-crate-pipeline/)

rust_crate_pipeline-1.4.0.dist-info/RECORD

@@ -1,19 +0,0 @@
-rust_crate_pipeline/__init__.py,sha256=NxD8_OEGHEHUN9EfJj2S1rRyZ0UMkiF20LNSMnjL9Uk,1939
-rust_crate_pipeline/__main__.py,sha256=fYgtPofuk4vkwiZ7ELP4GVMNj_QiKmZMSlvhzsNGuDs,155
-rust_crate_pipeline/ai_processing.py,sha256=sj-qPtIVLuuY_VoWoLbcGQ6_eS_giQyXIPyAGAWOCrs,24814
-rust_crate_pipeline/analysis.py,sha256=jcHHTBZ_zg5n4VGPXJYM7-NkNeL5hRdgvowkiim0onM,17663
-rust_crate_pipeline/config.py,sha256=xX4j_vgXaQxVI6Q3UmazzEzFdm6kLhpGbM2Of_fZS6k,2336
-rust_crate_pipeline/github_token_checker.py,sha256=_cyOiSYc1bCVczr6pUUJc_s822ic7Qi_IW3JtI_4C0w,3796
-rust_crate_pipeline/main.py,sha256=bemr27xpXIFYEyXtcCQfZpAQ5pPycyiRZKP8nj9kork,10111
-rust_crate_pipeline/network.py,sha256=MFtn_-9MRBUSehfjLboUBGOMk8gv2edjOjHCR_YEyGc,12677
-rust_crate_pipeline/pipeline.py,sha256=aOLuIpfvDbPDCvft8ppUa0vRiFVdiz2wltpi26ZJaes,22769
-rust_crate_pipeline/production_config.py,sha256=24YWT68Fo2Kl8v7Hn1WgqfPrikXma9VZEuEcMr7iDik,2282
-rust_crate_pipeline/version.py,sha256=4JXcc5UI7bkW_OwMSDTrt2YpSLowN-WFH11PYQDr_BQ,2614
-rust_crate_pipeline/utils/file_utils.py,sha256=IJOBBp6-w9pnCdqyGcRNwBph_iwI_zzULCdAULGFUy0,2097
-rust_crate_pipeline/utils/logging_utils.py,sha256=5-o6ohm38sH1ozjZWHPlm9Wj7yILiUzvMsLJDeu11lk,2350
-rust_crate_pipeline-1.4.0.dist-info/licenses/LICENSE,sha256=tpd4XNpbssrSx9-iErATOLrOh0ivNPfO2I5MAPUpats,1088
-rust_crate_pipeline-1.4.0.dist-info/METADATA,sha256=srt7t9sB9uJ70LF0jB9gvolb3qb4BLUtdBFYWfiAPDA,17474
-rust_crate_pipeline-1.4.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-rust_crate_pipeline-1.4.0.dist-info/entry_points.txt,sha256=9Rr_IRuFRIridXxUSdEJbB3ba0NnpEfKmknZXFdYRC0,70
-rust_crate_pipeline-1.4.0.dist-info/top_level.txt,sha256=GUdB7RyxHLhijQxui_KTy3B8p_L2APui9C6RYa0FuaE,20
-rust_crate_pipeline-1.4.0.dist-info/RECORD,,