ai-parrot 0.17.2__cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

parrot/handlers/crew/README_BOTMANAGER_PERSISTENCE.md

@@ -0,0 +1,303 @@
+# AgentsCrew Redis Persistence - Integrated into BotManager
+
+## Overview
+
+Redis persistence for AgentsCrew has been **fully integrated into BotManager**. Crews are automatically saved to and loaded from Redis, making persistence completely transparent to users of the API.
+
+## How It Works
+
+### Automatic Persistence
+
+The BotManager now handles all Redis persistence automatically:
+
+1. **On Startup** (`on_startup`):
+   - All crews are loaded from Redis into memory
+   - Similar to how bots are loaded from the database
+
+2. **When Creating/Updating a Crew** (`add_crew`):
+   - Crew is added to in-memory cache
+   - Automatically saved to Redis (key: `crew:{name}`)
+   - If Redis fails, crew remains in memory with a warning
+
+3. **When Getting a Crew** (`get_crew`):
+   - First checks in-memory cache
+   - If not found, loads from Redis
+   - Reconstructs the crew and caches it in memory
+
+4. **When Deleting a Crew** (`remove_crew`):
+   - Removes from in-memory cache
+   - Automatically deletes from Redis
+   - If Redis fails, crew is still removed from memory
+
+## Key Format
+
+Crews are stored in Redis with the following key structure:
+
+- **Crew Definition**: `crew:{name}` - Full CrewDefinition as JSON
+- **Crew List**: `crew:list` - Set of all crew names
+- **ID Mapping**: `crew:id:{crew_id}` - Maps crew_id to crew name
+- **Redis Database**: DB 2 (separate from conversation history on DB 3)
+
+## Usage
+
+### Via REST API
+
+The persistence is **completely transparent**. Just use the crew endpoints normally:
+
+```bash
+# Create a crew (automatically saved to Redis)
+curl -X PUT http://localhost:5000/api/v1/crew \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "research_team",
+    "execution_mode": "sequential",
+    "agents": [...]
+  }'
+
+# Get a crew (loads from Redis if not in memory)
+curl http://localhost:5000/api/v1/crew?name=research_team
+
+# Execute a crew
+curl -X POST http://localhost:5000/api/v1/crew \
+  -H "Content-Type: application/json" \
+  -d '{
+    "crew_id": "research_team",
+    "query": "Research AI trends"
+  }'
+
+# Delete a crew (removes from both memory and Redis)
+curl -X DELETE http://localhost:5000/api/v1/crew?name=research_team
+```
+
+### Programmatically
+
+If you're using BotManager directly in code:
+
+```python
+from parrot.manager.manager import BotManager
+from parrot.handlers.crew.models import CrewDefinition, AgentDefinition, ExecutionMode
+
+# Initialize manager
+bot_manager = BotManager()
+
+# Create crew definition
+crew_def = CrewDefinition(
+    name="my_crew",
+    execution_mode=ExecutionMode.SEQUENTIAL,
+    agents=[
+        AgentDefinition(
+            agent_id="agent1",
+            agent_class="BasicAgent",
+            name="Agent 1",
+            config={"model": "gpt-4"}
+        )
+    ]
+)
+
+# Create and add crew (automatically saved to Redis)
+crew = await bot_manager._create_crew_from_definition(crew_def)
+await bot_manager.add_crew(crew_def.name, crew, crew_def)
+
+# Get crew (loads from Redis if not in memory)
+crew_data = await bot_manager.get_crew("my_crew")
+if crew_data:
+    crew, crew_def = crew_data
+    print(f"Loaded: {crew_def.name}")
+
+# Remove crew (deletes from both memory and Redis)
+await bot_manager.remove_crew("my_crew")
+```
+
+## Configuration
+
+Redis configuration is in `parrot/conf.py`:
+
+```python
+REDIS_HOST = 'localhost'  # Default
+REDIS_PORT = 6379         # Default
+REDIS_DB = 2              # CrewRedis uses DB 2
+```
+
+## Startup Behavior
+
+When the application starts:
+
+1. **Loads Bots** from database (existing behavior)
+2. **Loads Crews** from Redis (new behavior)
+3. Both are ready for use immediately
+
+Example startup log:
+
+```
+INFO :: Bots loaded successfully. Total active bots: 5
+INFO Loading 3 crews from Redis...
+INFO Loaded crew 'research_team' with 3 agents in sequential mode
+INFO Loaded crew 'writing_team' with 2 agents in parallel mode
+INFO Loaded crew 'analysis_crew' with 4 agents in flow mode
+INFO :: Crews loaded successfully. Total active crews: 3
+```
+
+## Benefits
+
+### 1. **Zero Configuration**
+- No manual Redis calls needed
+- Works out of the box
+
+### 2. **Persistence Across Restarts**
+- Crews survive server restarts
+- Automatically restored on startup
+
+### 3. **Graceful Degradation**
+- If Redis is unavailable, crews still work in-memory
+- Warnings logged for Redis failures
+
+### 4. **Performance**
+- In-memory cache for fast access
+- Redis only accessed on cache miss
+
+## Architecture
+
+```
+┌─────────────────┐
+│   REST API      │
+│  CrewHandler    │
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│   BotManager    │
+│                 │
+│  ┌───────────┐  │
+│  │  Memory   │  │ ← Fast in-memory cache
+│  │  Cache    │  │
+│  └───────────┘  │
+│        │        │
+│        ▼        │
+│  ┌───────────┐  │
+│  │CrewRedis  │  │ ← Persistent storage
+│  └───────────┘  │
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│   Redis DB 2    │
+└─────────────────┘
+```
+
+## Error Handling
+
+The system handles Redis failures gracefully:
+
+```python
+# If Redis is down during startup
+WARNING Redis connection failed, skipping crew loading
+# → Application starts with empty crew cache
+
+# If Redis fails during save
+ERROR Failed to save crew 'my_crew' to Redis: Connection refused
+INFO Crew 'my_crew' registered in memory only (Redis persistence failed)
+# → Crew works normally but won't survive restart
+
+# If Redis fails during delete
+ERROR Failed to delete crew 'my_crew' from Redis: Connection refused
+INFO Crew 'my_crew' removed from memory only
+# → Crew removed from memory but remains in Redis (manual cleanup needed)
+```
+
+## Monitoring
+
+Check crew statistics:
+
+```python
+stats = bot_manager.get_crew_stats()
+print(stats)
+```
+
+Returns:
+
+```python
+{
+    'total_crews': 3,
+    'crews_by_mode': {
+        'sequential': 1,
+        'parallel': 1,
+        'flow': 1
+    },
+    'total_agents': 9,
+    'crews': [
+        {'name': 'research_team', 'crew_id': 'uuid...', 'mode': 'sequential', 'agent_count': 3},
+        {'name': 'writing_team', 'crew_id': 'uuid...', 'mode': 'parallel', 'agent_count': 2},
+        {'name': 'analysis_crew', 'crew_id': 'uuid...', 'mode': 'flow', 'agent_count': 4}
+    ]
+}
+```
+
+## Direct Redis Access (Optional)
+
+If you need to access Redis directly for debugging:
+
+```python
+# Access the Redis client from BotManager
+crew_redis = bot_manager.crew_redis
+
+# Check connection
+is_connected = await crew_redis.ping()
+
+# List all crews in Redis
+crew_names = await crew_redis.list_crews()
+
+# Get crew metadata
+metadata = await crew_redis.get_crew_metadata("my_crew")
+
+# Clear all crews (caution!)
+deleted_count = await crew_redis.clear_all_crews()
+```
+
+## Migration
+
+If you have crews in the old standalone CrewRedis system, they're automatically accessible via BotManager's `get_crew()` method, which will load them from Redis on first access.
+
+## Troubleshooting
+
+### Crew not found after restart
+
+Check if Redis is running:
+
+```bash
+redis-cli ping
+# Should return: PONG
+```
+
+Check if crew is in Redis:
+
+```bash
+redis-cli -n 2 KEYS "crew:*"
+```
+
+### Redis connection errors
+
+Verify Redis configuration in `parrot/conf.py`:
+
+```python
+REDIS_HOST = 'localhost'
+REDIS_PORT = 6379
+```
+
+Test connection:
+
+```python
+await bot_manager.crew_redis.ping()
+```
+
+## Summary
+
+Redis persistence for crews is now **fully automatic**:
+
+✅ Crews are saved to Redis when created
+✅ Crews are loaded from Redis on startup
+✅ Crews are loaded from Redis on first access
+✅ Crews are deleted from Redis when removed
+✅ All transparent to API users
+✅ Graceful fallback if Redis is unavailable
+
+No manual Redis operations needed - just use the CrewHandler API!

parrot/handlers/crew/README_REDIS_PERSISTENCE.md

@@ -0,0 +1,366 @@
+# Redis Persistence for AgentsCrew
+
+This module provides async-based Redis persistence for AgentsCrew definitions, allowing crews to be stored, retrieved, and managed across sessions.
+
+## Overview
+
+The `CrewRedis` class enables you to:
+- Save crew definitions to Redis with key format `crew:{name}`
+- Load crew definitions by name or crew_id
+- List all available crews
+- Update crew metadata
+- Delete crews
+- Manage multiple crews efficiently
+
+## Installation
+
+The Redis persistence layer requires the `redis` package with async support:
+
+```bash
+pip install redis
+```
+
+## Configuration
+
+Redis configuration is managed through `parrot/conf.py`:
+
+```python
+REDIS_HOST = 'localhost'  # Default
+REDIS_PORT = 6379         # Default
+REDIS_DB = 2              # CrewRedis uses DB 2 by default
+```
+
+## Basic Usage
+
+### Initialize CrewRedis
+
+```python
+from parrot.handlers.crew.redis_persistence import CrewRedis
+
+# Use default configuration
+crew_redis = CrewRedis()
+
+# Or specify custom Redis URL
+crew_redis = CrewRedis(redis_url="redis://localhost:6379/2")
+
+# Check connection
+is_connected = await crew_redis.ping()
+```
+
+### Save a Crew Definition
+
+```python
+from parrot.handlers.crew.models import CrewDefinition, AgentDefinition, ExecutionMode
+
+# Create a crew definition
+crew_def = CrewDefinition(
+    name="blog_writing_team",
+    description="A crew for researching and writing blog posts",
+    execution_mode=ExecutionMode.SEQUENTIAL,
+    agents=[
+        AgentDefinition(
+            agent_id="researcher",
+            agent_class="BasicAgent",
+            name="Content Researcher",
+            config={"model": "gpt-4", "temperature": 0.7},
+            tools=["web_search", "summarize"],
+            system_prompt="You are an expert researcher."
+        ),
+        AgentDefinition(
+            agent_id="writer",
+            agent_class="BasicAgent",
+            name="Blog Writer",
+            config={"model": "gpt-4", "temperature": 0.9},
+            tools=["write", "format_markdown"],
+            system_prompt="You are a creative blog writer."
+        )
+    ],
+    shared_tools=["calculator"],
+    metadata={"version": "1.0", "author": "content_team"}
+)
+
+# Save to Redis (key: crew:blog_writing_team)
+success = await crew_redis.save_crew(crew_def)
+```
+
+### Load a Crew Definition
+
+```python
+# Load by name (key: crew:blog_writing_team)
+crew = await crew_redis.load_crew("blog_writing_team")
+
+if crew:
+    print(f"Loaded crew: {crew.name}")
+    print(f"Agents: {len(crew.agents)}")
+    print(f"Execution mode: {crew.execution_mode}")
+
+# Load by crew_id
+crew = await crew_redis.load_crew_by_id("some-uuid-here")
+```
+
+### List All Crews
+
+```python
+# Get list of crew names
+crew_names = await crew_redis.list_crews()
+print(f"Available crews: {crew_names}")
+
+# Get all crew definitions
+all_crews = await crew_redis.get_all_crews()
+for crew in all_crews:
+    print(f"{crew.name}: {crew.description}")
+```
+
+### Check Crew Existence
+
+```python
+exists = await crew_redis.crew_exists("blog_writing_team")
+if exists:
+    print("Crew exists in Redis")
+```
+
+### Update Crew Metadata
+
+```python
+# Update metadata without loading full definition
+success = await crew_redis.update_crew_metadata(
+    "blog_writing_team",
+    {
+        "status": "active",
+        "runs": 10,
+        "last_run": "2025-12-12T10:00:00"
+    }
+)
+```
+
+### Get Crew Metadata
+
+```python
+# Get metadata without loading full crew definition
+metadata = await crew_redis.get_crew_metadata("blog_writing_team")
+if metadata:
+    print(f"Crew ID: {metadata['crew_id']}")
+    print(f"Agent count: {metadata['agent_count']}")
+    print(f"Created at: {metadata['created_at']}")
+```
+
+### Delete a Crew
+
+```python
+# Delete crew from Redis
+success = await crew_redis.delete_crew("blog_writing_team")
+```
+
+### Clean Up
+
+```python
+# Close Redis connection when done
+await crew_redis.close()
+```
+
+## Key Format
+
+The CrewRedis class uses the following key formats in Redis:
+
+- **Crew Definition**: `crew:{name}` - Stores the full CrewDefinition as JSON
+- **Crew List**: `crew:list` - Set of all crew names
+- **ID Mapping**: `crew:id:{crew_id}` - Maps crew_id to crew name for lookup by ID
+
+## Advanced Usage
+
+### Flow-Based Crews
+
+```python
+from parrot.handlers.crew.models import FlowRelation
+
+crew_def = CrewDefinition(
+    name="research_synthesis_team",
+    description="Parallel research with synthesis",
+    execution_mode=ExecutionMode.FLOW,
+    agents=[
+        AgentDefinition(
+            agent_id="coordinator",
+            name="Research Coordinator",
+            # ... config
+        ),
+        AgentDefinition(
+            agent_id="tech_researcher",
+            name="Tech Researcher",
+            # ... config
+        ),
+        AgentDefinition(
+            agent_id="synthesizer",
+            name="Synthesizer",
+            # ... config
+        )
+    ],
+    flow_relations=[
+        # Coordinator runs first, then researchers in parallel
+        FlowRelation(
+            source="coordinator",
+            target=["tech_researcher", "market_researcher"]
+        ),
+        # Synthesizer runs after both researchers complete
+        FlowRelation(
+            source=["tech_researcher", "market_researcher"],
+            target="synthesizer"
+        )
+    ]
+)
+
+await crew_redis.save_crew(crew_def)
+```
+
+### Bulk Operations
+
+```python
+# Get all crews
+all_crews = await crew_redis.get_all_crews()
+
+# Filter by execution mode
+sequential_crews = [
+    crew for crew in all_crews
+    if crew.execution_mode == ExecutionMode.SEQUENTIAL
+]
+
+# Clear all crews (use with caution!)
+deleted_count = await crew_redis.clear_all_crews()
+print(f"Deleted {deleted_count} crews")
+```
+
+## Integration with BotManager
+
+The CrewRedis class can be integrated with the existing BotManager for persistent storage:
+
+```python
+class BotManager:
+    def __init__(self):
+        self._crews = {}  # In-memory cache
+        self.crew_redis = CrewRedis()  # Persistent storage
+
+    async def create_crew(self, crew_def: CrewDefinition):
+        # Create crew instance
+        crew = AgentCrew(name=crew_def.name, agents=agents)
+
+        # Cache in memory
+        self._crews[crew_def.crew_id] = (crew, crew_def)
+
+        # Persist to Redis
+        await self.crew_redis.save_crew(crew_def)
+
+    async def get_crew(self, crew_id: str):
+        # Check memory cache first
+        if crew_id in self._crews:
+            return self._crews[crew_id]
+
+        # Load from Redis
+        crew_def = await self.crew_redis.load_crew_by_id(crew_id)
+        if crew_def:
+            # Reconstruct crew from definition
+            crew = self._create_crew_from_definition(crew_def)
+            self._crews[crew_id] = (crew, crew_def)
+            return (crew, crew_def)
+
+        return None
+```
+
+## Error Handling
+
+The CrewRedis class includes comprehensive error handling and logging:
+
+```python
+try:
+    crew = await crew_redis.load_crew("nonexistent_crew")
+    if crew is None:
+        print("Crew not found")
+except Exception as e:
+    print(f"Error loading crew: {e}")
+```
+
+All methods return appropriate values:
+- `save_crew()`: Returns `True` on success, `False` on failure
+- `load_crew()`: Returns `CrewDefinition` if found, `None` otherwise
+- `delete_crew()`: Returns `True` if deleted, `False` if not found
+- `list_crews()`: Returns empty list on error
+
+## Performance Considerations
+
+1. **Connection Pooling**: The Redis client automatically manages connection pooling
+2. **Serialization**: Uses JSON for efficient serialization/deserialization
+3. **Caching**: Consider implementing an in-memory cache layer for frequently accessed crews
+4. **Batch Operations**: Use `get_all_crews()` for bulk loading instead of multiple individual loads
+
+## Testing
+
+Run the built-in test suite:
+
+```bash
+# Test the persistence layer
+python parrot/handlers/crew/redis_persistence.py
+
+# Run the comprehensive examples
+python examples/crew/redis_persistence_example.py
+```
+
+## Database Selection
+
+By default, CrewRedis uses Redis database 2 to avoid conflicts with other Redis-based storage:
+
+- **DB 0**: General purpose (default)
+- **DB 1**: Configuration cache
+- **DB 2**: Crew persistence (CrewRedis)
+- **DB 3**: Conversation history
+- **DB 4**: Services
+
+## Troubleshooting
+
+### Redis Connection Failed
+
+```python
+if not await crew_redis.ping():
+    print("Check Redis server is running: redis-cli ping")
+```
+
+### Serialization Errors
+
+Ensure all crew data types are JSON-serializable. The module automatically handles datetime serialization.
+
+### Key Conflicts
+
+If using custom key prefixes, ensure they don't conflict with existing Redis keys:
+
+```python
+crew_redis = CrewRedis(key_prefix="myapp_crew")
+```
+
+## Example Scripts
+
+See `/examples/crew/redis_persistence_example.py` for comprehensive examples including:
+- Basic persistence
+- Flow-based crews
+- Metadata updates
+- Bulk operations
+
+## API Reference
+
+### CrewRedis Class
+
+#### Methods
+
+- `__init__(redis_url=None, key_prefix="crew", db=2)`
+- `save_crew(crew: CrewDefinition) -> bool`
+- `load_crew(name: str) -> Optional[CrewDefinition]`
+- `load_crew_by_id(crew_id: str) -> Optional[CrewDefinition]`
+- `delete_crew(name: str) -> bool`
+- `list_crews() -> List[str]`
+- `crew_exists(name: str) -> bool`
+- `get_all_crews() -> List[CrewDefinition]`
+- `get_crew_metadata(name: str) -> Optional[Dict[str, Any]]`
+- `update_crew_metadata(name: str, metadata: Dict[str, Any]) -> bool`
+- `ping() -> bool`
+- `close() -> None`
+- `clear_all_crews() -> int`
+
+## License
+
+This module is part of the ai-parrot project and follows the same MIT license.