portacode 0.3.4.dev0__py3-none-any.whl → 1.4.11.dev0__py3-none-any.whl

testing_framework/README.md

@@ -0,0 +1,334 @@
+# Modular Testing Framework
+
+A comprehensive, modular testing framework designed specifically for Portacode projects that combines CLI connection management with Playwright-based web automation testing.
+
+## 🚀 Features
+
+- **Modular Architecture**: Organize tests by categories and tags
+- **CLI Integration**: Automatically connects to Portacode CLI in background threads  
+- **Playwright Automation**: Full web browser automation with comprehensive recording
+- **Selective Execution**: Run all tests, specific categories, tags, or individual tests
+- **Comprehensive Recording**: Screenshots, videos, network logs, console output, and traces
+- **Rich Reporting**: HTML and JSON reports with detailed test results
+
+## 📋 Quick Start
+
+### 1. Installation
+
+```bash
+# Install testing framework dependencies
+pip install -r requirements-testing.txt
+
+# Install Playwright browsers
+python -m playwright install
+```
+
+### 2. Environment Setup
+
+Copy the example environment file and configure it:
+
+```bash
+cp .env.example .env
+```
+
+Edit `.env` with your configuration:
+
+```env
+TEST_BASE_URL=http://192.168.1.188:8001/
+TEST_USERNAME=your_username
+TEST_PASSWORD=your_password
+```
+
+### 3. Run Tests
+
+```bash
+# List all available tests
+python -m testing_framework.cli list-tests
+
+# Run all tests
+python -m testing_framework.cli run-all
+
+# Run tests by category
+python -m testing_framework.cli run-category smoke
+
+# Run tests by tags
+python -m testing_framework.cli run-tags login authentication
+
+# Run specific tests
+python -m testing_framework.cli run-tests login_flow_test device_connection_test
+
+# Run tests matching a pattern
+python -m testing_framework.cli run-pattern "login.*"
+```
+
+## 📁 Project Structure
+
+```
+testing_framework/
+├── __init__.py                 # Framework exports
+├── cli.py                     # Command-line interface
+└── core/
+    ├── __init__.py
+    ├── base_test.py           # Base test class and categories
+    ├── cli_manager.py         # CLI connection management
+    ├── playwright_manager.py  # Playwright session management
+    ├── test_discovery.py      # Test discovery system
+    └── runner.py              # Test runner with reporting
+
+test_modules/
+├── __init__.py
+├── test_login_flow.py         # Login flow test example
+├── test_device_connection.py  # Device connection test
+├── test_ui_navigation.py      # UI navigation test
+└── test_performance_check.py  # Performance test example
+```
+
+## 🔧 Framework Architecture
+
+### Test Execution Flow
+
+1. **CLI Connection**: Each test starts by connecting to Portacode CLI in a background thread
+2. **Playwright Session**: Browser session starts and automatically logs in using provided credentials
+3. **Test Execution**: Your test logic runs with access to both CLI and browser automation
+4. **Recording**: Everything is recorded - screenshots, videos, network traffic, console logs
+5. **Cleanup**: Resources are properly cleaned up and recordings are saved
+
+### Test Categories
+
+- `SMOKE`: Basic functionality tests
+- `INTEGRATION`: Cross-system integration tests  
+- `UI`: User interface tests
+- `API`: API endpoint tests
+- `PERFORMANCE`: Performance and load tests
+- `SECURITY`: Security-focused tests
+- `CUSTOM`: Custom test categories
+
+## ✍️ Writing Custom Tests
+
+### Basic Test Structure
+
+```python
+from testing_framework.core.base_test import BaseTest, TestResult, TestCategory
+
+class MyCustomTest(BaseTest):
+    def __init__(self):
+        super().__init__(
+            name="my_custom_test",
+            category=TestCategory.SMOKE,
+            description="Description of what this test does",
+            tags=["tag1", "tag2"]
+        )
+    
+    async def run(self) -> TestResult:
+        # Your test logic here
+        page = self.playwright_manager.page
+        
+        # Take screenshots
+        await self.playwright_manager.take_screenshot("test_step_1")
+        
+        # Interact with the page
+        await page.click("button")
+        
+        # Check CLI connection
+        if self.cli_manager.is_connection_active():
+            # CLI is connected and working
+            pass
+            
+        # Return test result
+        return TestResult(self.name, True, "Test passed!")
+    
+    async def setup(self):
+        # Optional setup code
+        pass
+    
+    async def teardown(self):
+        # Optional cleanup code
+        pass
+```
+
+### Available Managers
+
+#### CLI Manager (`self.cli_manager`)
+
+- `is_connection_active()`: Check if CLI connection is active
+- `get_connection_info()`: Get connection details
+- `get_log_content()`: Get CLI output logs
+
+#### Playwright Manager (`self.playwright_manager`)
+
+- `page`: Direct access to Playwright page object
+- `take_screenshot(name)`: Take named screenshot
+- `log_action(type, details)`: Log custom actions
+- `get_recordings_info()`: Get recording information
+
+## 📊 Test Output
+
+### Directory Structure
+
+```
+test_results/
+└── run_20241201_143022/
+    ├── summary.json           # Test run summary
+    ├── report.html           # HTML report
+    ├── test_run.log          # Framework logs
+    ├── cli_logs/             # CLI output logs
+    │   └── test_name_timestamp_cli.log
+    └── recordings/           # Playwright recordings
+        └── test_name_timestamp/
+            ├── recording.webm    # Video recording
+            ├── trace.zip        # Playwright trace
+            ├── network.har      # Network logs
+            ├── console.log      # Browser console
+            ├── actions.json     # Logged actions
+            ├── screenshots/     # All screenshots
+            └── summary.json     # Recording summary
+```
+
+### HTML Report
+
+The framework generates comprehensive HTML reports with:
+- Test run statistics and timeline
+- Pass/fail status for each test
+- Screenshots and recordings links
+- Error messages and logs
+- Performance metrics
+
+## 🎛️ Configuration Options
+
+### Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `TEST_BASE_URL` | Application URL | `http://192.168.1.188:8001/` |
+| `TEST_USERNAME` | Login username | Required |
+| `TEST_PASSWORD` | Login password | Required |
+| `TEST_BROWSER` | Browser type | `chromium` |
+| `TEST_HEADLESS` | Headless mode | `false` |
+| `TEST_RESULTS_DIR` | Results directory | `test_results` |
+| `TEST_RECORDINGS_DIR` | Recordings directory | `test_recordings` |
+| `TEST_LOGS_DIR` | Logs directory | `test_results` |
+
+### Command Line Options
+
+```bash
+# Enable debug logging
+python -m testing_framework.cli --debug run-all
+
+# All commands support debug mode
+python -m testing_framework.cli --debug list-tests
+```
+
+## 🔍 Debugging and Troubleshooting
+
+### Common Issues
+
+1. **CLI Connection Fails**
+   - Check that `portacode.cli` module is available
+   - Verify CLI credentials and connectivity
+   - Check CLI logs in `cli_logs/` directory
+
+2. **Playwright Login Fails**
+   - Verify `TEST_USERNAME` and `TEST_PASSWORD` in `.env`
+   - Check login form selectors in `playwright_manager.py`
+   - Review screenshots in recordings directory
+
+3. **Tests Not Discovered**
+   - Ensure test files are in `test_modules/` or `tests/` directories
+   - Test files must start with `test_` or end with `_test.py`
+   - Test classes must inherit from `BaseTest`
+
+### Debug Mode
+
+Run with debug logging to see detailed execution information:
+
+```bash
+python -m testing_framework.cli --debug run-all
+```
+
+### Manual Debugging
+
+Access the recordings directory to inspect:
+- Video recordings of test execution
+- Screenshots at each step
+- Network traffic logs
+- Browser console output
+- Detailed action logs
+
+## 🤝 Contributing
+
+### Adding New Test Categories
+
+1. Add new category to `TestCategory` enum in `base_test.py`
+2. Update CLI command choices in `cli.py`
+
+### Extending Managers
+
+- **CLI Manager**: Add new CLI interaction methods
+- **Playwright Manager**: Add new browser automation helpers
+- **Test Runner**: Add new execution modes or reporting formats
+
+### Example: Custom Test Category
+
+```python
+# In base_test.py
+class TestCategory(Enum):
+    # ... existing categories ...
+    ACCESSIBILITY = "accessibility"
+
+# In your test
+class AccessibilityTest(BaseTest):
+    def __init__(self):
+        super().__init__(
+            name="accessibility_test",
+            category=TestCategory.ACCESSIBILITY,
+            description="Check accessibility compliance",
+            tags=["a11y", "compliance"]
+        )
+```
+
+## 📈 Advanced Usage
+
+### Programmatic Test Execution
+
+```python
+import asyncio
+from testing_framework.core.runner import TestRunner
+from testing_framework.core.base_test import TestCategory
+
+async def run_custom_suite():
+    runner = TestRunner()
+    
+    # Run specific category
+    results = await runner.run_tests_by_category(TestCategory.SMOKE)
+    
+    # Process results
+    if results['statistics']['failed'] > 0:
+        print("Some tests failed!")
+    
+    return results
+
+# Run it
+results = asyncio.run(run_custom_suite())
+```
+
+### Custom Reporting
+
+```python
+from testing_framework.core.runner import TestRunner
+
+class CustomRunner(TestRunner):
+    async def _generate_custom_report(self, results):
+        # Your custom reporting logic
+        pass
+```
+
+## 🔒 Security Considerations
+
+- Store credentials in `.env` file, never in code
+- Add `.env` to `.gitignore`
+- Use environment-specific credential management
+- Review recordings before sharing (may contain sensitive data)
+
+## 📝 License
+
+This testing framework is part of the Portacode project and follows the same license terms.

testing_framework/__init__.py

@@ -0,0 +1,17 @@
+"""
+Modular Testing Framework for Portacode
+
+A comprehensive testing framework that supports:
+- CLI connection management with background threading
+- Playwright-based web automation testing
+- Modular test organization with categories
+- Selective test execution
+- Comprehensive recording and logging
+"""
+
+from .core.base_test import BaseTest, TestCategory
+from .core.runner import TestRunner
+from .core.cli_manager import CLIManager
+from .core.playwright_manager import PlaywrightManager
+
+__all__ = ['BaseTest', 'TestCategory', 'TestRunner', 'CLIManager', 'PlaywrightManager']

testing_framework/cli.py

@@ -0,0 +1,326 @@
+"""Command-line interface for the testing framework."""
+
+import asyncio
+import click
+import logging
+import sys
+import os
+from pathlib import Path
+
+# Load environment variables from .env file if available
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+from .core.hierarchical_runner import HierarchicalTestRunner
+from .core.base_test import TestCategory
+
+# Use hierarchical runner as the default
+TestRunner = HierarchicalTestRunner
+
+
+def setup_logging(debug: bool = False):
+    """Setup logging configuration - logs only to files, not console."""
+    level = logging.DEBUG if debug else logging.INFO
+    
+    # Only log to files, not to console
+    # Create a null handler to prevent any console output from framework logs
+    logging.basicConfig(
+        level=level,
+        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+        handlers=[logging.NullHandler()]
+    )
+    
+    # If debug is enabled, we can optionally add a file handler here
+    if debug:
+        # Create debug log file in current directory
+        debug_handler = logging.FileHandler('framework_debug.log')
+        debug_handler.setFormatter(logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+        ))
+        logging.getLogger().addHandler(debug_handler)
+
+
+@click.group()
+@click.option('--debug', is_flag=True, help='Enable debug logging')
+@click.pass_context
+def cli(ctx, debug):
+    """Modular Testing Framework for Portacode"""
+    ctx.ensure_object(dict)
+    ctx.obj['debug'] = debug
+    setup_logging(debug)
+
+
+@cli.command()
+@click.pass_context
+async def list_tests(ctx):
+    """List all available tests."""
+    runner = TestRunner()
+    info = runner.list_available_tests()
+    
+    click.echo(f"📋 Found {click.style(str(info['total_tests']), fg='green')} tests")
+    click.echo(f"Categories: {click.style(', '.join([cat.value for cat in info['categories']]), fg='blue')}")
+    if info['tags']:
+        click.echo(f"Tags: {click.style(', '.join(info['tags']), fg='cyan')}")
+    
+    click.echo("\n📝 Available Tests:")
+    for name, test_info in info['tests'].items():
+        click.echo(f"  • {click.style(name, fg='yellow')}")
+        click.echo(f"    Category: {click.style(test_info['category'], fg='blue')}")
+        click.echo(f"    Description: {test_info['description']}")
+        if test_info['tags']:
+            click.echo(f"    Tags: {click.style(', '.join(test_info['tags']), fg='cyan')}")
+        click.echo()
+
+
+@cli.command()
+@click.option('--clear', is_flag=True, help='Clear test_results directory before running tests')
+@click.pass_context
+async def run_all(ctx, clear):
+    """Run all available tests with dependency resolution."""
+    if clear:
+        click.echo("🗑️  Clearing test_results directory...")
+    click.echo("🚀 Running all tests with dependency resolution...")
+    click.echo("🔗 Starting shared CLI connection...", nl=False)
+    runner = TestRunner(clear_results=clear)
+    results = await runner.run_all_tests(_create_progress_callback())
+    _print_results(results)
+
+
+@cli.command()
+@click.argument('category', type=click.Choice([cat.value for cat in TestCategory]))
+@click.option('--clear', is_flag=True, help='Clear test_results directory before running tests')
+@click.pass_context
+async def run_category(ctx, category, clear):
+    """Run tests in a specific category with dependency resolution."""
+    cat_enum = TestCategory(category)
+    if clear:
+        click.echo("🗑️  Clearing test_results directory...")
+    click.echo(f"🎯 Running {category} tests with dependency resolution...")
+    click.echo("🔗 Starting shared CLI connection...", nl=False)
+    runner = TestRunner(clear_results=clear)
+    results = await runner.run_tests_by_category(cat_enum, _create_progress_callback())
+    _print_results(results)
+
+
+@cli.command()
+@click.argument('tags', nargs=-1, required=True)
+@click.option('--clear', is_flag=True, help='Clear test_results directory before running tests')
+@click.pass_context
+async def run_tags(ctx, tags, clear):
+    """Run tests with specific tags."""
+    if clear:
+        click.echo("🗑️  Clearing test_results directory...")
+    click.echo(f"🏷️  Running tests with tags: {', '.join(tags)}...")
+    click.echo("🔗 Starting shared CLI connection...", nl=False)
+    runner = TestRunner(clear_results=clear)
+    results = await runner.run_tests_by_tags(set(tags), _create_progress_callback())
+    _print_results(results)
+
+
+@cli.command()
+@click.argument('names', nargs=-1, required=True)
+@click.option('--clear', is_flag=True, help='Clear test_results directory before running tests')
+@click.pass_context
+async def run_tests(ctx, names, clear):
+    """Run specific tests by name."""
+    if clear:
+        click.echo("🗑️  Clearing test_results directory...")
+    click.echo(f"📝 Running tests: {', '.join(names)}...")
+    click.echo("🔗 Starting shared CLI connection...", nl=False)
+    runner = TestRunner(clear_results=clear)
+    results = await runner.run_tests_by_names(list(names), _create_progress_callback())
+    _print_results(results)
+
+
+@cli.command()
+@click.argument('pattern')
+@click.option('--clear', is_flag=True, help='Clear test_results directory before running tests')
+@click.pass_context
+async def run_pattern(ctx, pattern, clear):
+    """Run tests matching a name pattern."""
+    if clear:
+        click.echo("🗑️  Clearing test_results directory...")
+    click.echo(f"🔍 Running tests matching pattern: {pattern}...")
+    click.echo("🔗 Starting shared CLI connection...", nl=False)
+    runner = TestRunner(clear_results=clear)
+    results = await runner.run_tests_by_pattern(pattern, _create_progress_callback())
+    _print_results(results)
+
+
+@cli.command()
+@click.option('--clear', is_flag=True, help='Clear test_results directory before running tests')
+@click.pass_context
+async def run_hierarchical(ctx, clear):
+    """Run all tests with hierarchical dependency resolution."""
+    if clear:
+        click.echo("🗑️  Clearing test_results directory...")
+    click.echo("🚀 Running tests with dependency resolution...")
+    click.echo("📋 Analyzing test dependencies...")
+    
+    runner = HierarchicalTestRunner(clear_results=clear)
+    results = await runner.run_all_tests(_create_progress_callback())
+    _print_results(results)
+    
+    # Show dependency information
+    if results.get('results'):
+        skipped = [r for r in results['results'] if 'Skipped:' in r.get('message', '')]
+        if skipped:
+            click.echo(f"\n⏭️  Skipped Tests ({len(skipped)}):")
+            for result in skipped:
+                click.echo(f"  • {result['test_name']}: {result['message']}")
+
+
+@cli.command() 
+@click.argument('names', nargs=-1, required=True)
+@click.pass_context
+async def run_hierarchical_tests(ctx, names):
+    """Run specific tests with dependency resolution."""
+    click.echo(f"📝 Running tests with dependencies: {', '.join(names)}...")
+    click.echo("📋 Analyzing test dependencies...")
+    
+    runner = HierarchicalTestRunner()
+    results = await runner.run_tests_by_names(list(names), _create_progress_callback())
+    _print_results(results)
+
+
+def _create_progress_callback():
+    """Create a progress callback for clean console output."""
+    cli_connected_shown = False
+    
+    def progress_callback(event, test, current, total, result=None):
+        nonlocal cli_connected_shown
+        
+        if event == 'start':
+            # Show CLI connected message only once
+            if not cli_connected_shown:
+                click.echo("\r🔗 Shared CLI connection established ✅")
+                cli_connected_shown = True
+            # Clean one-line output for test start  
+            click.echo(f"[{current}/{total}] 🔄 {test.name}", nl=False)
+        elif event == 'complete' and result:
+            # Clear the line and show result
+            click.echo(f"\r[{current}/{total}] {'✅' if result.success else '❌'} {test.name} ({result.duration:.2f}s)", nl=True)
+            if not result.success and result.message:
+                click.echo(f"    └─ {click.style(result.message, fg='red')}")
+    
+    return progress_callback
+
+
+def _print_results(results):
+    """Print test results summary with stylish stats formatting."""
+    if not results.get('results'):
+        click.echo("❌ No tests were run")
+        return
+        
+    stats = results['statistics']
+    duration = results['run_info']['duration']
+    
+    # Stylish header
+    click.echo("\n" + "="*60)
+    click.echo(f"{'📊 TEST RESULTS SUMMARY':^60}")
+    click.echo("="*60)
+    
+    # Main stats with better formatting
+    click.echo(f"  📋 Total Tests:    {click.style(str(stats['total_tests']), fg='cyan', bold=True)}")
+    click.echo(f"  ⏱️  Total Duration: {click.style(f'{duration:.2f}s', fg='blue', bold=True)}")
+    click.echo(f"  ✅ Passed:        {click.style(str(stats['passed']), fg='green', bold=True)}")
+    click.echo(f"  ❌ Failed:        {click.style(str(stats['failed']), fg='red', bold=True)}")
+    
+    # Success rate with color coding
+    success_rate_text = f"{stats['success_rate']:.2f}%"
+    success_rate_color = 'green' if stats['success_rate'] > 80 else 'yellow' if stats['success_rate'] > 50 else 'red'
+    click.echo(f"  📈 Success Rate:  {click.style(success_rate_text, fg=success_rate_color, bold=True)}")
+    
+    # Individual test stats with timing
+    click.echo(f"\n{'⚡ PERFORMANCE BREAKDOWN':^60}")
+    click.echo("-"*60)
+    
+    # Show timing stats for each test
+    for result in results['results']:
+        status_icon = "✅" if result['success'] else "❌"
+        test_name = result['test_name'].replace('_test', '').replace('_', ' ').title()
+        duration_text = f"{result['duration']:.2f}s"
+        duration_color = 'green' if result['duration'] < 5 else 'yellow' if result['duration'] < 10 else 'red'
+        
+        click.echo(f"  {status_icon} {test_name:<30} {click.style(duration_text, fg=duration_color)}")
+        
+        # Show additional stats if available
+        if result.get('artifacts') and isinstance(result['artifacts'], dict):
+            # Check both direct artifacts and nested timings/stats
+            all_stats = {}
+            
+            # Add direct artifacts
+            for key, value in result['artifacts'].items():
+                if isinstance(value, (int, float)):
+                    all_stats[key] = value
+            
+            # Add nested timings
+            if 'timings' in result['artifacts'] and isinstance(result['artifacts']['timings'], dict):
+                for key, value in result['artifacts']['timings'].items():
+                    if isinstance(value, (int, float)):
+                        all_stats[key] = value
+            
+            # Add nested stats
+            if 'stats' in result['artifacts'] and isinstance(result['artifacts']['stats'], dict):
+                for key, value in result['artifacts']['stats'].items():
+                    if isinstance(value, (int, float)):
+                        all_stats[key] = value
+            
+            # Display all stats
+            for key, value in all_stats.items():
+                # Format based on the value and key name
+                if 'ms' in key.lower() or value > 100:
+                    formatted_value = f"{value:.2f}ms"
+                else:
+                    formatted_value = f"{value:.2f}s"
+                
+                # Clean up key name for display
+                display_key = key.replace('_', ' ').replace('time ms', 'time').title()
+                click.echo(f"    └─ {display_key}: {click.style(formatted_value, fg='blue')}")
+    
+    click.echo(f"\n📂 Results saved to: {click.style(results['run_info']['run_directory'], fg='blue', underline=True)}")
+    
+    # Show failed tests summary if any
+    failed_tests = [r for r in results['results'] if not r['success']]
+    if failed_tests:
+        click.echo(f"\n{'❌ FAILED TESTS DETAILS':^60}")
+        click.echo("-"*60)
+        for result in failed_tests:
+            click.echo(f"  • {click.style(result['test_name'], fg='red', bold=True)}")
+            if result.get('message'):
+                # Handle multi-line error messages
+                message_lines = result['message'].split('\n')
+                for i, line in enumerate(message_lines):
+                    prefix = "    └─ " if i == 0 else "       "
+                    click.echo(f"{prefix}{click.style(line, fg='red')}")
+                
+                # Add note about trace viewer
+                click.echo(f"       {click.style('🔍 Trace viewer should open automatically for investigation', fg='yellow')}")
+    
+    # Summary footer
+    click.echo("="*60)
+
+
+# Async command wrapper
+def async_command(f):
+    def wrapper(*args, **kwargs):
+        return asyncio.run(f(*args, **kwargs))
+    return wrapper
+
+
+# Convert async commands
+list_tests.callback = async_command(list_tests.callback)
+run_all.callback = async_command(run_all.callback)
+run_category.callback = async_command(run_category.callback)
+run_tags.callback = async_command(run_tags.callback)
+run_tests.callback = async_command(run_tests.callback)
+run_pattern.callback = async_command(run_pattern.callback)
+run_hierarchical.callback = async_command(run_hierarchical.callback)
+run_hierarchical_tests.callback = async_command(run_hierarchical_tests.callback)
+
+
+if __name__ == '__main__':
+    cli()

testing_framework/core/__init__.py

@@ -0,0 +1 @@
+"""Core testing framework components."""