campfirevalley 1.0.0__tar.gz

campfirevalley-1.0.0/PKG-INFO

@@ -0,0 +1,704 @@
+Metadata-Version: 2.4
+Name: campfirevalley
+Version: 1.0.0
+Summary: A sophisticated multi-agent AI system for collaborative problem-solving through specialized AI campfires
+Home-page: https://github.com/MikeHibbert/pyCamperValley
+Author: Mike Hibbert
+Author-email: mike@hibbertitsolutions.co.uk
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: campfires>=0.4.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: redis>=4.5.0
+Requires-Dist: cryptography>=41.0.0
+Requires-Dist: aiofiles>=23.0.0
+Requires-Dist: prometheus-client>=0.17.0
+Requires-Dist: structlog>=23.0.0
+Requires-Dist: jsonschema>=4.17.0
+Requires-Dist: click>=8.1.0
+Requires-Dist: asyncio-mqtt>=0.13.0
+Requires-Dist: psycopg2-binary>=2.9.0
+Requires-Dist: elasticsearch>=8.0.0
+Requires-Dist: grafana-api>=1.0.3
+Requires-Dist: pytest>=7.0.0
+Requires-Dist: pytest-asyncio>=0.21.0
+Requires-Dist: pytest-cov>=4.0.0
+Requires-Dist: black>=23.0.0
+Requires-Dist: flake8>=6.0.0
+Requires-Dist: mypy>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=6.0.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=1.2.0; extra == "docs"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CampfireValley
+
+A Python library that extends the Campfires framework by introducing the concept of "valleys" as interconnected, secure communities of campfires. The library manages docks as gateways for inter-valley communication, handles riverboat (torch) exchanges via MCP, and provides default campfires for loading/offloading, security sanitization, and justice enforcement.
+
+## 🚀 Project Status
+
+**Phase 2 Implementation: COMPLETED** ✅
+
+CampfireValley is now **~95% complete** with comprehensive enterprise-grade features implemented and tested. The system provides a complete distributed torch processing framework.
+
+### ✅ Phase 1 Components (Core Infrastructure)
+
+- **🔐 Key Manager**: Complete AES-256 and RSA key management system
+- **🔄 Redis MCP Broker**: Full pub/sub messaging with connection management  
+- **📦 Torch Serialization**: Enhanced with Redis transport and MCP communication
+- **🏭 Dockmaster Campfire**: Complete with Loader, Router, and Packer campers
+- **🏗️ Core Architecture**: Interfaces, models, and base classes
+
+### ✅ Phase 2 Components (Enterprise Features)
+
+- **🛡️ VALI Services**: Complete security scanning and validation framework
+- **⚖️ Justice System**: Advanced policy enforcement and violation handling
+- **🔧 Specialist Campfires**: Domain-specific processing units (Sanitizer, Validator, Router)
+- **🌐 Advanced Routing**: Multi-hop routing with load balancing and failover
+- **📊 Monitoring & Logging**: Comprehensive metrics, alerts, and structured logging
+- **⚙️ Configuration Management**: Multi-environment config with validation and encryption
+- **💾 Hierarchical Storage**: Multi-tier storage with compression and deduplication
+
+### 🔄 Future Enhancements
+
+- **Web Interface**: Management dashboard and monitoring UI
+- **Distributed Consensus**: Multi-valley coordination protocols
+- **Machine Learning**: Intelligent routing and threat detection
+
+## Features
+
+### Core Infrastructure (Phase 1)
+- **Valley Management**: Self-contained instances hosting multiple campfires
+- **Dock Gateways**: Secure entry points for inter-valley communication
+- **MCP Integration**: Redis-based message communication protocol
+- **Key Management**: AES-256 encryption and RSA digital signatures
+- **GitHub Actions-style Configuration**: Familiar YAML configuration format
+
+### Enterprise Features (Phase 2)
+- **🛡️ VALI Security Services**: 
+  - Content validation and signature verification
+  - Enhanced security scanning with threat detection
+  - Service registry and coordination
+- **⚖️ Justice System**: 
+  - Policy-based access control and enforcement
+  - Violation detection and automated responses
+  - Rate limiting and abuse prevention
+- **🔧 Specialist Campfires**: 
+  - Sanitizer: Content cleaning and threat removal
+  - Validator: Data integrity and format validation
+  - Router: Intelligent routing decisions
+- **🌐 Advanced Routing**: 
+  - Multi-hop routing with path optimization
+  - Smart load balancing algorithms
+  - Automatic failover and health checking
+- **📊 Monitoring & Logging**: 
+  - Real-time metrics collection and alerting
+  - Structured logging with multiple handlers
+  - Performance monitoring and health checks
+- **⚙️ Configuration Management**: 
+  - Multi-environment configuration (dev/prod)
+  - Schema validation and encryption
+  - Hot-reload and change tracking
+- **💾 Hierarchical Storage**: 
+  - Multi-tier storage (hot/warm/cold/archive)
+  - Data deduplication and compression
+  - Intelligent lifecycle management
+
+## 🎯 Demos & Examples
+
+CampfireValley includes comprehensive demos showcasing real-world AI agent collaboration workflows. These demos demonstrate the complete system capabilities from idea generation to technical implementation.
+
+### 🚀 Marketing Team Demo
+
+The marketing team demo showcases a complete AI-driven workflow where marketing strategists generate innovative ideas and collaborate with development teams to create technical implementations.
+
+#### Demo Architecture
+
+```
+┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
+│  Marketing      │    │   Development    │    │   Generated     │
+│  Strategist     │───▶│   Team API       │───▶│   Report        │
+│  (AI Agent)     │    │  (Dockerized)    │    │   (JSON/HTML)   │
+└─────────────────┘    └──────────────────┘    └─────────────────┘
+```
+
+#### Quick Demo Execution
+
+**Option 1: Simplified Demo (Recommended)**
+```bash
+# Start Docker containers
+docker-compose up -d
+
+# Run the simplified marketing demo
+python simple_marketing_demo.py
+```
+
+**Option 2: Full Valley Demo**
+```bash
+# Start Docker containers
+docker-compose up -d
+
+# Run the complete valley demo
+python demo_marketing_team.py
+```
+
+#### Demo Workflow
+
+1. **Idea Generation**: Marketing strategist AI generates innovative business ideas
+2. **Strategic Analysis**: Each idea is analyzed for market potential and feasibility
+3. **Development Requests**: Ideas are formatted and sent to the development team API
+4. **Technical Analysis**: Development team provides detailed technical requirements
+5. **Report Generation**: Complete workflow results are compiled into comprehensive reports
+
+#### Expected Output
+
+The demo generates:
+- **3 Marketing Ideas**: E-commerce Innovation, SaaS Solution, Mobile App Concept
+- **Strategic Analysis**: Market research, target audience, competitive analysis
+- **Technical Requirements**: Architecture, features, UX considerations
+- **Development Analysis**: Detailed technical implementation plans
+- **Comprehensive Report**: JSON and HTML formats with complete workflow results
+
+#### Demo Files
+
+- `simple_marketing_demo.py`: Streamlined demo bypassing MCP complexities
+- `demo_marketing_team.py`: Full valley implementation with MCP integration
+- `development_team_server.py`: Dockerized development team API service
+- `docker-compose.yml`: Container orchestration for demo services
+
+### 🐳 Docker Container Services
+
+The demo utilizes several containerized services:
+
+| Service | Container | Port | Purpose |
+|---------|-----------|------|---------|
+| Development Team | `campfire-development-team` | 8080 | Technical analysis API |
+| Redis MCP Broker | `campfire-redis` | 6379 | Message communication |
+| Ollama LLM | `campfire-ollama` | 11434 | Local AI model serving |
+| Prometheus | `campfire-prometheus` | 9090 | Monitoring and metrics |
+
+#### Container Status Check
+```bash
+# View running containers
+docker ps
+
+# Check container logs
+docker logs campfire-development-team
+docker logs campfire-redis
+```
+
+### 🔧 Troubleshooting
+
+#### Common Issues and Solutions
+
+**1. MCP Broker Connection Issues**
+```bash
+# Symptom: Demo hangs at "Subscribing to MCP channels"
+# Solution: Use simplified demo
+python simple_marketing_demo.py
+```
+
+**2. Docker Container Issues**
+```bash
+# Restart all containers
+docker-compose down
+docker-compose up -d
+
+# Check container health
+docker ps
+```
+
+**3. Development Team API 404 Errors**
+```bash
+# Verify development team container is running
+docker logs campfire-development-team
+
+# Check API endpoint
+curl http://localhost:8080/api/develop_website
+```
+
+**4. Port Conflicts**
+```bash
+# Check port usage
+netstat -an | findstr :8080
+netstat -an | findstr :6379
+
+# Modify ports in docker-compose.yml if needed
+```
+
+#### Demo Variations
+
+**Simplified Workflow (No MCP)**
+- Bypasses MCP broker subscription issues
+- Direct API communication
+- Faster execution
+- Recommended for demonstrations
+
+**Full Valley Workflow (With MCP)**
+- Complete CampfireValley architecture
+- MCP broker integration
+- Inter-valley communication
+- Production-like environment
+
+### 📊 Demo Reports
+
+Demo execution generates detailed reports:
+
+**JSON Report Structure**:
+```json
+{
+  "execution_summary": {
+    "ideas_generated": 3,
+    "dev_requests_sent": 3,
+    "successful_responses": 3,
+    "execution_time": "45.2 seconds"
+  },
+  "marketing_ideas": [...],
+  "development_analysis": [...],
+  "workflow_metrics": {...}
+}
+```
+
+**Report Locations**:
+- `simplified_marketing_report_YYYYMMDD_HHMMSS.json`
+- `marketing_team_report.html` (if HTML generation enabled)
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install campfirevalley
+```
+
+### Basic Usage
+
+1. **Create a valley configuration**:
+```bash
+campfirevalley create-config MyValley --output manifest.yaml
+```
+
+2. **Start a valley**:
+```bash
+campfirevalley start MyValley --manifest manifest.yaml
+```
+
+3. **Programmatic usage**:
+```python
+import asyncio
+from campfirevalley import Valley
+
+async def main():
+    # Create and start a valley
+    valley = Valley("MyValley", "./manifest.yaml")
+    await valley.start()
+    
+    # Valley is now running and can communicate with other valleys
+    print(f"Valley '{valley.name}' is running")
+    
+    # Stop the valley
+    await valley.stop()
+
+asyncio.run(main())
+```
+
+## 🏗️ Phase 1 Implementation Details
+
+### Key Manager (`campfirevalley.key_manager`)
+- **AES-256 Encryption**: Secure payload encryption/decryption
+- **RSA Digital Signatures**: Torch signing and verification
+- **Key Rotation**: Community-based key management
+- **Secure Storage**: Encrypted key storage in `.secrets/`
+
+### Redis MCP Broker (`campfirevalley.mcp`)
+- **Pub/Sub Messaging**: Redis-based inter-valley communication
+- **Connection Management**: Robust connection pooling and error handling
+- **Message Routing**: Channel-based message distribution
+- **Async Support**: Full asyncio compatibility
+
+### Enhanced Torch Model (`campfirevalley.models`)
+- **Redis Serialization**: Optimized for Redis transport
+- **MCP Envelopes**: Structured message format
+- **Compression**: Automatic compression for large payloads
+- **Routing Channels**: Smart channel determination
+
+### Dockmaster Campfire (`campfirevalley.campfires.dockmaster`)
+- **LoaderCamper**: Torch validation and unpacking
+- **RouterCamper**: Intelligent routing decisions
+- **PackerCamper**: Transport preparation and packaging
+- **Pipeline Processing**: Complete torch processing workflow
+
+## 🏗️ Phase 2 Implementation Details
+
+### VALI Services (`campfirevalley.vali`)
+- **Service Registry**: Centralized service discovery and management
+- **Content Validator**: Deep content analysis and validation
+- **Signature Verifier**: Cryptographic signature verification
+- **Security Scanner**: Advanced threat detection with multiple engines
+- **Coordinator**: Service orchestration and lifecycle management
+
+### Justice System (`campfirevalley.justice`)
+- **Policy Engine**: Rule-based access control and enforcement
+- **Enforcement Engine**: Automated violation response and remediation
+- **Violation Detection**: Real-time monitoring and threat identification
+- **Rate Limiting**: Configurable rate limiting and abuse prevention
+- **Audit Trail**: Complete violation and enforcement logging
+
+### Specialist Campfires (`campfirevalley.specialist_campfires`)
+- **SanitizerCampfire**: Content sanitization with multiple levels
+- **ValidatorCampfire**: Data validation with custom rules
+- **RouterCampfire**: Intelligent routing with strategy selection
+- **Configurable Rules**: Custom sanitization, validation, and routing rules
+- **Pipeline Integration**: Seamless integration with torch processing
+
+### Advanced Routing (`campfirevalley.routing`)
+- **Route Optimization**: Intelligent path selection and optimization
+- **Load Balancing**: Multiple algorithms (round-robin, weighted, least-connections)
+- **Health Checking**: Continuous endpoint health monitoring
+- **Failover Strategies**: Automatic failover with multiple strategies
+- **Metrics Collection**: Real-time routing performance metrics
+
+### Monitoring & Logging (`campfirevalley.monitoring`)
+- **Metrics System**: Comprehensive metrics collection and aggregation
+- **Alert Management**: Configurable alerts with multiple severity levels
+- **Structured Logging**: JSON-based logging with multiple handlers
+- **Performance Monitoring**: Real-time performance tracking
+- **Health Checking**: System-wide health monitoring
+
+### Configuration Management (`campfirevalley.config_manager`)
+- **Multi-Environment**: Separate configs for development, production, etc.
+- **Schema Validation**: JSON Schema-based configuration validation
+- **Encryption Support**: Sensitive configuration data encryption
+- **Hot Reload**: Runtime configuration updates without restart
+- **Change Tracking**: Complete configuration change history
+
+### Hierarchical Storage (`campfirevalley.hierarchical_storage`)
+- **Multi-Tier Storage**: Hot, warm, cold, and archive tiers
+- **Data Lifecycle**: Intelligent data movement between tiers
+- **Deduplication**: Content-based deduplication to save space
+- **Compression**: Configurable compression algorithms
+- **Storage Optimization**: Automated storage optimization and cleanup
+
+### Usage Example
+
+```python
+import asyncio
+from campfirevalley import Valley, CampfireKeyManager, RedisMCPBroker
+from campfirevalley.campfires import DockmasterCampfire
+from campfirevalley.models import Torch
+
+async def main():
+    # Initialize components
+    key_manager = CampfireKeyManager("MyValley")
+    mcp_broker = RedisMCPBroker("redis://localhost:6379")
+    dockmaster = DockmasterCampfire(mcp_broker)
+    
+    # Start services
+    await mcp_broker.connect()
+    await dockmaster.start()
+    
+    # Create and process a torch
+    torch = Torch(
+        id="example_001",
+        sender_valley="MyValley",
+        target_address="TargetValley:dockmaster/loader",
+        payload={"message": "Hello from CampfireValley!"},
+        signature="example_signature"
+    )
+    
+    # Process through Dockmaster pipeline
+    response = await dockmaster.process_torch(torch)
+    print(f"Processed torch: {response}")
+    
+    # Cleanup
+    await dockmaster.stop()
+    await mcp_broker.disconnect()
+
+asyncio.run(main())
+```
+
+## 🔧 Configuration-Driven Architecture
+
+CampfireValley implements a comprehensive configuration-driven approach that enables flexible, maintainable, and environment-specific deployments without code changes.
+
+### Core Configuration Philosophy
+
+The system follows these key principles:
+- **Declarative Configuration**: Define what you want, not how to achieve it
+- **Environment Separation**: Clean separation between dev, staging, and production
+- **Schema Validation**: All configurations are validated against JSON schemas
+- **Hot Reload**: Runtime configuration updates without service restart
+- **Encryption Support**: Sensitive data is automatically encrypted
+
+### Configuration Hierarchy
+
+CampfireValley uses a multi-layered configuration system:
+
+1. **Base Configuration**: Core system defaults
+2. **Environment Configuration**: Environment-specific overrides (dev/prod)
+3. **Valley Configuration**: Valley-specific settings
+4. **Runtime Configuration**: Dynamic updates during operation
+
+### GitHub Actions-Style YAML Format
+
+CampfireValley uses familiar GitHub Actions-style YAML configuration:
+
+```yaml
+name: "MyValley"
+version: "1.0"
+
+env:
+  dock_mode: "public"
+  security_level: "standard"
+  auto_create_dock: true
+
+campfires:
+  visible: ["helper", "processor"]
+  hidden: ["internal"]
+
+community:
+  discovery: true
+  trusted_valleys: ["FriendValley"]
+
+# Advanced configuration sections
+security:
+  vali:
+    enabled: true
+    scan_level: "comprehensive"
+    threat_detection: true
+  
+  justice:
+    enabled: true
+    policy_enforcement: "strict"
+    rate_limiting:
+      requests_per_minute: 100
+      burst_limit: 20
+
+routing:
+  strategy: "intelligent"
+  load_balancing: "weighted_round_robin"
+  health_checks:
+    enabled: true
+    interval: 30
+    timeout: 5
+
+storage:
+  hierarchical:
+    enabled: true
+    tiers:
+      hot: { retention: "7d", compression: false }
+      warm: { retention: "30d", compression: "lz4" }
+      cold: { retention: "1y", compression: "gzip" }
+      archive: { retention: "7y", compression: "bzip2" }
+
+monitoring:
+  metrics:
+    enabled: true
+    collection_interval: 10
+  
+  logging:
+    level: "INFO"
+    format: "json"
+    handlers: ["console", "file", "redis"]
+  
+  alerts:
+    enabled: true
+    channels: ["email", "slack"]
+```
+
+### Environment-Specific Configuration
+
+Create separate configuration files for different environments:
+
+**config/dev.yaml**:
+```yaml
+extends: "base.yaml"
+
+env:
+  security_level: "development"
+  debug: true
+
+monitoring:
+  logging:
+    level: "DEBUG"
+
+security:
+  vali:
+    scan_level: "basic"
+```
+
+**config/prod.yaml**:
+```yaml
+extends: "base.yaml"
+
+env:
+  security_level: "maximum"
+  debug: false
+
+security:
+  vali:
+    scan_level: "comprehensive"
+    threat_detection: true
+  
+  justice:
+    policy_enforcement: "strict"
+
+monitoring:
+  alerts:
+    enabled: true
+    severity_threshold: "warning"
+```
+
+### Configuration Management Features
+
+#### 1. Schema Validation
+All configurations are validated against JSON schemas:
+
+```python
+from campfirevalley.config_manager import ConfigManager
+
+# Load and validate configuration
+config_manager = ConfigManager()
+config = config_manager.load_config("config/prod.yaml")
+# Automatically validates against schema
+```
+
+#### 2. Encryption Support
+Sensitive configuration data is automatically encrypted:
+
+```yaml
+database:
+  host: "db.example.com"
+  username: "admin"
+  password: "!encrypted:AES256:base64encodeddata"  # Auto-encrypted
+```
+
+#### 3. Hot Reload
+Update configurations without restarting services:
+
+```python
+# Configuration changes are automatically detected and applied
+config_manager.watch_for_changes()
+```
+
+#### 4. Environment Variables
+Override any configuration value with environment variables:
+
+```bash
+export CAMPFIREVALLEY_SECURITY_LEVEL="maximum"
+export CAMPFIREVALLEY_MONITORING_LOGGING_LEVEL="ERROR"
+```
+
+### Configuration Sections Reference
+
+#### Core Settings
+- `name`: Valley identifier
+- `version`: Configuration version
+- `env`: Environment variables and basic settings
+
+#### Security Configuration
+- `security.vali`: VALI security services settings
+- `security.justice`: Justice system and policy enforcement
+- `security.encryption`: Encryption and key management
+
+#### Routing Configuration
+- `routing.strategy`: Routing algorithm selection
+- `routing.load_balancing`: Load balancing configuration
+- `routing.health_checks`: Health monitoring settings
+
+#### Storage Configuration
+- `storage.hierarchical`: Multi-tier storage settings
+- `storage.compression`: Compression algorithms
+- `storage.retention`: Data retention policies
+
+#### Monitoring Configuration
+- `monitoring.metrics`: Metrics collection settings
+- `monitoring.logging`: Logging configuration
+- `monitoring.alerts`: Alert management
+
+### Best Practices
+
+1. **Use Environment Inheritance**: Extend base configurations for environments
+2. **Validate Early**: Always validate configurations during startup
+3. **Encrypt Secrets**: Use the built-in encryption for sensitive data
+4. **Monitor Changes**: Enable configuration change tracking
+5. **Test Configurations**: Validate configurations in CI/CD pipelines
+
+### Configuration Examples
+
+See the `examples/` directory for complete configuration examples:
+- `examples/basic_valley.yaml`: Simple valley setup
+- `examples/enterprise_valley.yaml`: Full enterprise configuration
+- `examples/development.yaml`: Development environment setup
+- `examples/production.yaml`: Production-ready configuration
+
+## Architecture
+
+- **Valley**: Main container managing campfires and infrastructure
+- **Dock**: Gateway for inter-valley communication
+- **Campfire**: Individual AI agent containers
+- **Torch**: Message format for inter-valley communication
+- **Party Box**: Storage system for attachments and payloads
+- **MCP Broker**: Redis-based message routing
+
+## Development
+
+### Requirements
+
+- Python 3.8+
+- Redis server (for MCP broker)
+- PyYAML, Pydantic, cryptography
+
+### Running Tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+## Additional Documentation
+
+- **[Demo Guide](DEMO_GUIDE.md)**: Comprehensive guide for executing demos and examples
+- **[Deployment Guide](DEPLOYMENT.md)**: Detailed deployment instructions and Docker setup
+- **[Troubleshooting Guide](TROUBLESHOOTING.md)**: Solutions for common issues and MCP broker problems
+
+## Support
+
+If you encounter issues:
+
+1. Check the [Troubleshooting Guide](TROUBLESHOOTING.md) for common problems
+2. Review the [Demo Guide](DEMO_GUIDE.md) for step-by-step execution instructions
+3. Use the simplified demo (`simple_marketing_demo.py`) to bypass MCP broker issues
+4. Check Docker container logs: `docker-compose logs [service-name]`
+
+## License
+
+MIT License - see LICENSE file for details.