kolega-code 0.1.0__py3-none-any.whl

kolega_code/sandbox/README.md

@@ -0,0 +1,200 @@
+# Cloud Sandbox Implementation
+
+This directory contains the cloud sandbox implementation for running AI agents in isolated environments using E2B.
+
+## Architecture Overview
+
+The sandbox system provides a pluggable architecture that allows agents to run in cloud-based isolated environments instead of locally. The implementation uses E2B as the sandbox provider but is designed to support other providers in the future.
+
+### Key Components
+
+1. **Base Interfaces** (`base.py`)
+   - `SandboxConfig`: Configuration for sandbox creation (git URL, branch, environment variables)
+   - `ProjectManifest`: Project configuration for dependencies and setup
+   - `SandboxManager`: Abstract base class for sandbox management
+
+2. **E2B Implementation** (`sandbox_e2b.py`)
+   - `E2BSandboxManager`: Concrete implementation using E2B sandboxes
+   - Handles Git authentication using GitLab group access tokens
+   - Manages sandbox lifecycle (create, destroy, commit, push)
+   - Mounts workspace files from S3 for agent access
+
+3. **Sandbox-Aware Services**
+   - `SandboxFileSystem`: FileSystem implementation that operates within sandbox
+   - `SandboxTerminalManager`: Terminal manager for executing commands in sandbox
+   - `SandboxBrowserManager`: Browser manager placeholder (not yet implemented)
+
+4. **Utilities** (`utils.py`)
+   - Helper functions for common sandbox operations
+   - Git status parsing and diff generation
+   - Project test execution
+
+## How It Works
+
+1. **Host Job Integration**: When a host application job starts and sandbox mode is enabled:
+   - Creates an E2B sandbox with the workspace's Git repository
+   - Clones the repository using GitLab group access token
+   - Mounts workspace files from S3 as read-only
+   - Installs dependencies if a manifest file exists
+   - Injects sandbox-aware services into the agent
+
+2. **Agent Execution**: The agent runs with:
+   - Filesystem operations redirected to sandbox
+   - Terminal commands executed in sandbox
+   - Access to uploaded workspace files at `/home/user/workspace/kolega-project-files`
+   - All changes isolated from local environment
+
+3. **Git Workflow**: After agent completion:
+   - Detects modified files using `git status`
+   - Commits changes with descriptive message
+   - Pushes changes back to GitLab repository
+
+4. **Workspace Files**: Users can upload files through the UI which are:
+   - Stored in S3 at `<bucket>/<workspace_id>/<filename>`
+   - Automatically mounted in sandboxes at `/home/user/workspace/kolega-project-files`
+   - Available as read-only to prevent accidental modifications
+   - Excluded from git tracking via `.gitignore`
+
+## Configuration
+
+### Environment Variables
+
+```bash
+# Enable sandbox mode
+USE_SANDBOX=true
+
+# Sandbox provider (currently only 'e2b' supported)
+SANDBOX_PROVIDER=e2b
+
+# E2B API key from https://e2b.dev
+E2B_API_KEY=your_e2b_api_key
+
+# E2B template (must use custom template with s3fs)
+E2B_TEMPLATE=your_custom_template_id
+
+# GitLab group access token with read/write permissions
+GITLAB_GROUP_ACCESS_TOKEN=glpat-xxxxxxxxxxxxxxxxxxxx
+
+# S3 Configuration for workspace files
+S3_BUCKET_NAME=your-kolega-files-bucket
+AWS_ACCESS_KEY_ID=your-access-key
+AWS_SECRET_ACCESS_KEY=your-secret-key
+AWS_REGION=us-east-1  # Optional, defaults to us-east-1
+```
+
+### E2B Template Requirements
+
+The E2B template must have s3fs installed. See `backend/e2b-template/` for the custom template that includes:
+- Base E2B code interpreter
+- s3fs for mounting S3 buckets
+- Proper permissions setup
+
+Build and deploy the template:
+```bash
+cd backend/e2b-template
+e2b template build
+# Update E2B_TEMPLATE env var with the new template ID
+```
+
+### Project Manifest
+
+Projects can include a `.kolega-manifest.yaml` file in the repository root:
+
+```yaml
+name: my-project
+runtime: node:18  # or python:3.11, etc.
+
+# Optional: Commands to install dependencies
+install_commands:
+  - npm install
+
+# Optional: Commands to run before install
+environment_setup:
+  - npm config set registry https://registry.npmjs.org/
+
+# Optional: Development server command (runs automatically in background)
+dev_server_command: npm run dev
+
+# Optional: Test commands
+test_commands:
+  - npm test
+
+# Optional: Build command
+build_command: npm run build
+```
+
+#### Dev Server Auto-Start
+
+When a `dev_server_command` is specified in the manifest, it will automatically start in the background after dependencies are installed. This is useful for:
+
+- React/Vue/Angular development servers
+- API servers (Express, FastAPI, etc.)
+- Documentation servers
+- Any long-running development process
+
+Example configurations:
+
+**React App:**
+```yaml
+name: my-react-app
+runtime: node:18
+install_commands:
+  - npm install
+dev_server_command: npm start -- --port 9001
+
+```
+
+**Python FastAPI:**
+```yaml
+name: my-api
+runtime: python:3.11
+install_commands:
+  - pip install -r requirements.txt
+dev_server_command: uvicorn main:app --reload --host 0.0.0.0 --port 9001
+
+```
+
+**Next.js App:**
+```yaml
+name: my-nextjs-app
+runtime: node:18
+install_commands:
+  - npm install
+dev_server_command: npm run dev -- -p 9001
+```
+
+**Minimal Project (no dependencies):**
+```yaml
+name: simple-static-site
+runtime: node:18
+# No install_commands needed for a simple static site
+dev_server_command: python -m http.server 9001
+```
+
+## Usage
+
+The sandbox is automatically used when:
+1. `USE_SANDBOX=true` in environment
+2. Workspace has a GitLab repository (`gitlab_project_url` is set)
+3. Agent job is started by the host application
+
+No code changes are required in agents - they continue using the standard FileSystem and TerminalManager interfaces.
+
+### Accessing Workspace Files
+
+Agents can access uploaded workspace files at:
+```python
+workspace_files = "/home/user/workspace/kolega-project-files"
+
+# List files
+import os
+files = os.listdir(workspace_files)
+
+# Read a file
+with open(f"{workspace_files}/document.pdf", "rb") as f:
+    content = f.read()
+```
+
+## Testing
+
+The implementation includes comprehensive tests:

kolega_code/sandbox/__init__.py

@@ -0,0 +1,21 @@
+"""Sandbox services for cloud-based agent execution."""
+
+from .base import SandboxConfig, ProjectManifest, SandboxManager
+from .filesystem import SandboxFileSystem
+from .async_filesystem import AsyncSandboxFileSystem
+from .terminal import SandboxTerminalManager
+from .browser import SandboxBrowserManager
+from .local import LocalSandboxManager
+from .utils import get_modified_files_from_sandbox
+
+__all__ = [
+    "SandboxConfig",
+    "ProjectManifest",
+    "SandboxManager",
+    "SandboxFileSystem",
+    "AsyncSandboxFileSystem",
+    "SandboxTerminalManager",
+    "SandboxBrowserManager",
+    "LocalSandboxManager",
+    "get_modified_files_from_sandbox",
+]

kolega_code/sandbox/async_filesystem.py

@@ -0,0 +1,475 @@
+"""Async FileSystem implementation for sandbox environments."""
+
+import os
+import base64
+from pathlib import Path
+from typing import Any, Dict, Iterator, List, Optional, Union, BinaryIO
+from contextlib import contextmanager
+from datetime import datetime
+import asyncio
+import nest_asyncio
+
+from ..services.file_system import FileSystem
+
+
+class AsyncSandboxFileSystem(FileSystem):
+    """FileSystem implementation that operates within an async sandbox.
+
+    This class provides a synchronous interface to an async sandbox by using
+    nest_asyncio to allow running async operations in sync contexts.
+    """
+
+    def __init__(self, sandbox: Any, root_path: str = "/home/user/workspace"):
+        """
+        Initialize async sandbox filesystem.
+
+        Args:
+            sandbox: The async sandbox instance (e.g., AsyncE2B Sandbox)
+            root_path: Root path within the sandbox
+        """
+        self.sandbox = sandbox
+        self.root_path = root_path
+        self._nest_asyncio_applied = False
+
+    def _run_async(self, coro):
+        """Run an async coroutine in a sync context using nest_asyncio."""
+        try:
+            # Try to get the current running loop
+            loop = asyncio.get_running_loop()
+
+            # Apply nest_asyncio only once and only when needed
+            if not self._nest_asyncio_applied:
+                try:
+                    # Only apply if we're not in a uvloop context
+                    if not hasattr(loop, "__module__") or "uvloop" not in loop.__module__:
+                        nest_asyncio.apply()
+                        self._nest_asyncio_applied = True
+                except:
+                    # If patching fails, we'll try alternative approaches
+                    pass
+
+        except RuntimeError:
+            # No running loop, create a new one
+            loop = asyncio.new_event_loop()
+            asyncio.set_event_loop(loop)
+            # Apply nest_asyncio for the new loop
+            if not self._nest_asyncio_applied:
+                nest_asyncio.apply()
+                self._nest_asyncio_applied = True
+
+        # Use nest_asyncio to run the coroutine if it was applied
+        if self._nest_asyncio_applied:
+            return loop.run_until_complete(coro)
+        else:
+            # If we couldn't apply nest_asyncio (e.g., uvloop),
+            # we need to use a different approach
+            import concurrent.futures
+            import threading
+
+            # Create a new event loop in a separate thread
+            def run_in_new_loop():
+                # Lazy import to avoid circular dependency
+                from kolega_code.sandbox.event_loop import cleanup_event_loop
+
+                new_loop = asyncio.new_event_loop()
+                asyncio.set_event_loop(new_loop)
+                try:
+                    return new_loop.run_until_complete(coro)
+                finally:
+                    # Proper cleanup following asyncio.Runner pattern
+                    cleanup_event_loop(new_loop)
+
+            with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
+                future = executor.submit(run_in_new_loop)
+                return future.result()
+
+    def _resolve_path(self, path: str) -> str:
+        """Resolve path relative to root."""
+        if os.path.isabs(path):
+            return path
+        if path == ".":
+            return self.root_path
+        return os.path.join(self.root_path, path)
+
+    # Synchronous methods wrapping async E2B API
+    def open(self, path: str, mode: str = "r", encoding: Optional[str] = None) -> Any:
+        """Open is not directly supported in sandbox - use read/write methods instead."""
+        raise NotImplementedError("Direct file handles not supported in sandbox. Use read_text/write_text instead.")
+
+    def read_text(self, path: str, encoding: str = "utf-8") -> str:
+        """Read text from file."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _read():
+                return await self.sandbox.files.read(full_path)
+
+            content = self._run_async(_read())
+            if isinstance(content, bytes):
+                return content.decode(encoding)
+            return content
+        except Exception as e:
+            raise FileNotFoundError(f"Could not read file {path}: {e}")
+
+    def read_bytes(self, path: str) -> bytes:
+        """Read bytes from file."""
+        full_path = self._resolve_path(path)
+        try:
+            # For binary data, use base64 encoding via shell commands
+            # to avoid E2B files API text encoding issues
+            async def _read_bytes():
+                result = await self.sandbox.commands.run(f"base64 {full_path}")
+                if result.exit_code != 0:
+                    raise FileNotFoundError(f"Could not read file {path}")
+                # Decode the base64 content
+                return base64.b64decode(result.stdout.strip())
+
+            return self._run_async(_read_bytes())
+
+        except Exception as e:
+            raise FileNotFoundError(f"Could not read file {path}: {e}")
+
+    def write_text(self, path: str, content: str, encoding: str = "utf-8") -> None:
+        """Write text to file."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _write():
+                # Ensure parent directory exists
+                parent_dir = os.path.dirname(full_path)
+                if parent_dir != self.root_path:
+                    await self.sandbox.commands.run(f"mkdir -p {parent_dir}")
+
+                await self.sandbox.files.write(full_path, content)
+
+            self._run_async(_write())
+        except Exception as e:
+            raise OSError(f"Could not write file {path}: {e}")
+
+    def write_bytes(self, path: str, content: bytes) -> None:
+        """Write bytes to file."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _write_bytes():
+                # Ensure parent directory exists
+                parent_dir = os.path.dirname(full_path)
+                if parent_dir != self.root_path:
+                    await self.sandbox.commands.run(f"mkdir -p {parent_dir}")
+
+                # For binary data, use base64 encoding and write via shell commands
+                # to avoid E2B files API text encoding issues
+                encoded_content = base64.b64encode(content).decode("ascii")
+
+                # Write the base64 encoded content and decode it
+                result = await self.sandbox.commands.run(f"echo '{encoded_content}' | base64 -d > {full_path}")
+
+                if result.exit_code != 0:
+                    raise OSError(f"Failed to write binary file {path}: {result.stderr}")
+
+            self._run_async(_write_bytes())
+
+        except Exception as e:
+            raise OSError(f"Could not write file {path}: {e}")
+
+    def exists(self, path: str) -> bool:
+        """Check if path exists."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _exists():
+                result = await self.sandbox.commands.run(f"test -e {full_path}")
+                return result.exit_code == 0
+
+            return self._run_async(_exists())
+        except:
+            return False
+
+    def is_file(self, path: str) -> bool:
+        """Check if path is a file."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _is_file():
+                result = await self.sandbox.commands.run(f"test -f {full_path}")
+                return result.exit_code == 0
+
+            return self._run_async(_is_file())
+        except:
+            return False
+
+    def is_dir(self, path: str) -> bool:
+        """Check if path is a directory."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _is_dir():
+                result = await self.sandbox.commands.run(f"test -d {full_path}")
+                return result.exit_code == 0
+
+            return self._run_async(_is_dir())
+        except:
+            return False
+
+    def stat(self, path: str) -> Dict[str, Any]:
+        """Get file statistics."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _stat():
+                # Use stat command to get file info
+                result = await self.sandbox.commands.run(f"stat -c '%s %Y %Z' {full_path}")
+                if result.exit_code != 0:
+                    raise FileNotFoundError(f"File not found: {path}")
+
+                size, mtime, ctime = result.stdout.strip().split()
+
+                return {
+                    "size": int(size),
+                    "modified_time": int(mtime),
+                    "created_time": int(ctime),
+                    "is_file": await self._async_is_file(full_path),
+                    "is_directory": await self._async_is_dir(full_path),
+                }
+
+            return self._run_async(_stat())
+        except Exception as e:
+            raise OSError(f"Could not stat {path}: {e}")
+
+    async def _async_is_file(self, full_path: str) -> bool:
+        """Async helper to check if path is a file."""
+        try:
+            result = await self.sandbox.commands.run(f"test -f {full_path}")
+            return result.exit_code == 0
+        except Exception:
+            # E2B throws exception on non-zero exit codes
+            return False
+
+    async def _async_is_dir(self, full_path: str) -> bool:
+        """Async helper to check if path is a directory."""
+        try:
+            result = await self.sandbox.commands.run(f"test -d {full_path}")
+            return result.exit_code == 0
+        except Exception:
+            # E2B throws exception on non-zero exit codes
+            return False
+
+    def mkdir(self, path: str, parents: bool = False, exist_ok: bool = False) -> None:
+        """Create directory."""
+        full_path = self._resolve_path(path)
+
+        # Check if directory already exists
+        if self.exists(path):
+            if not exist_ok:
+                raise FileExistsError(f"Directory already exists: {path}")
+            return
+
+        try:
+
+            async def _mkdir():
+                if parents:
+                    result = await self.sandbox.commands.run(f"mkdir -p {full_path}")
+                else:
+                    result = await self.sandbox.commands.run(f"mkdir {full_path}")
+
+                if result.exit_code != 0:
+                    raise OSError(f"Failed to create directory {path}: {result.stderr}")
+
+            self._run_async(_mkdir())
+        except Exception as e:
+            raise OSError(f"Could not create directory {path}: {e}")
+
+    def remove(self, path: str, missing_ok: bool = False) -> None:
+        """Remove file."""
+        full_path = self._resolve_path(path)
+
+        if not self.exists(path):
+            if not missing_ok:
+                raise FileNotFoundError(f"File not found: {path}")
+            return
+
+        try:
+
+            async def _remove():
+                result = await self.sandbox.commands.run(f"rm -f {full_path}")
+                if result.exit_code != 0:
+                    raise OSError(f"Failed to remove file {path}: {result.stderr}")
+
+            self._run_async(_remove())
+        except Exception as e:
+            raise OSError(f"Could not remove file {path}: {e}")
+
+    def rmdir(self, path: str) -> None:
+        """Remove empty directory."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _rmdir():
+                result = await self.sandbox.commands.run(f"rmdir {full_path}")
+                if result.exit_code != 0:
+                    raise OSError(f"Failed to remove directory {path}: {result.stderr}")
+
+            self._run_async(_rmdir())
+        except Exception as e:
+            raise OSError(f"Could not remove directory {path}: {e}")
+
+    def rmtree(self, path: str) -> None:
+        """Remove directory tree."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _rmtree():
+                result = await self.sandbox.commands.run(f"rm -rf {full_path}")
+                if result.exit_code != 0:
+                    raise OSError(f"Failed to remove directory tree {path}: {result.stderr}")
+
+            self._run_async(_rmtree())
+        except Exception as e:
+            raise OSError(f"Could not remove directory tree {path}: {e}")
+
+    def listdir(self, path: str) -> List[str]:
+        """List directory contents."""
+        full_path = self._resolve_path(path)
+
+        if not self.is_dir(path):
+            raise NotADirectoryError(f"Not a directory: {path}")
+
+        try:
+
+            async def _listdir():
+                result = await self.sandbox.commands.run(f"ls -1 {full_path}")
+                if result.exit_code != 0:
+                    raise FileNotFoundError(f"Directory not found: {path}")
+
+                if not result.stdout.strip():
+                    return []
+
+                return [line.strip() for line in result.stdout.strip().split("\n") if line.strip()]
+
+            return self._run_async(_listdir())
+        except Exception as e:
+            raise OSError(f"Could not list directory {path}: {e}")
+
+    def iterdir(self, path: str) -> Iterator[str]:
+        """Iterate directory contents."""
+        return iter(self.listdir(path))
+
+    def glob(self, pattern: str) -> List[str]:
+        """Find paths matching pattern."""
+        try:
+
+            async def _glob():
+                # Handle different types of glob patterns
+                if pattern.startswith("**/"):
+                    # Recursive pattern like **/*.py or **/*
+                    remaining_pattern = pattern[3:]  # Remove '**/'
+
+                    if remaining_pattern == "*":
+                        # Pattern is **/* - find all files and directories recursively
+                        result = await self.sandbox.commands.run(
+                            f"cd {self.root_path} && find . -mindepth 1 2>/dev/null | sed 's|^./||' | sort"
+                        )
+                    else:
+                        # Pattern like **/*.py - find files matching pattern recursively
+                        if "*" in remaining_pattern or "?" in remaining_pattern:
+                            # Use find with -name for pattern matching
+                            result = await self.sandbox.commands.run(
+                                f"cd {self.root_path} && find . -name '{remaining_pattern}' 2>/dev/null | sed 's|^./||' | sort"
+                            )
+                        else:
+                            # Exact filename search recursively
+                            result = await self.sandbox.commands.run(
+                                f"cd {self.root_path} && find . -name '{remaining_pattern}' 2>/dev/null | sed 's|^./||' | sort"
+                            )
+
+                elif "*" in pattern or "?" in pattern:
+                    # Simple glob pattern like *.py or test*.txt
+                    if "/" in pattern:
+                        # Pattern has directory component like subdir/*.txt
+                        parent_dir = os.path.dirname(pattern)
+                        filename_pattern = os.path.basename(pattern)
+                        result = await self.sandbox.commands.run(
+                            f"cd {self.root_path} && find {parent_dir} -maxdepth 1 -name '{filename_pattern}' 2>/dev/null | sort"
+                        )
+                    else:
+                        # Simple pattern like *.py in current directory
+                        result = await self.sandbox.commands.run(
+                            f"cd {self.root_path} && find . -maxdepth 1 -name '{pattern}' 2>/dev/null | sed 's|^./||' | sort"
+                        )
+
+                else:
+                    # No wildcards - check if exact path exists
+                    if self.exists(pattern):
+                        return [pattern]
+                    else:
+                        return []
+
+                # Process the result
+                if result.exit_code == 0 and result.stdout.strip():
+                    paths = [line.strip() for line in result.stdout.strip().split("\n") if line.strip()]
+                    # Filter out empty strings and current directory
+                    return [path for path in paths if path and path != "."]
+                else:
+                    return []
+
+            return self._run_async(_glob())
+
+        except Exception as e:
+            # If any error occurs, return empty list to match expected behavior
+            return []
+
+    def is_binary_file(self, path: str) -> bool:
+        """Check if file is binary."""
+        full_path = self._resolve_path(path)
+        try:
+
+            async def _is_binary():
+                # Use file command to detect binary
+                result = await self.sandbox.commands.run(f"file -b --mime {full_path}")
+                return result.exit_code == 0 and "charset=binary" in result.stdout
+
+            return self._run_async(_is_binary())
+        except:
+            return False
+
+    def get_name(self, path: str) -> str:
+        """Get basename of path."""
+        return os.path.basename(path)
+
+    def get_suffix(self, path: str) -> str:
+        """Get file extension."""
+        return os.path.splitext(path)[1]
+
+    def get_parent(self, path: str) -> str:
+        """Get parent directory."""
+        parent = os.path.dirname(path)
+        # Return "." for root-level files to match LocalFileSystem behavior
+        return parent if parent else "."
+
+    def get_parents(self, path: str) -> List[str]:
+        """Get all parent directories."""
+        parents = []
+        current = os.path.dirname(path)
+        while current and current != "/":
+            parents.append(current)
+            current = os.path.dirname(current)
+        return parents
+
+    def relative_to(self, path: str, other: str) -> str:
+        """Get relative path."""
+        return os.path.relpath(path, other)
+
+    def join_path(self, *parts: str) -> str:
+        """Join path components."""
+        return os.path.join(*parts)
+
+    def is_absolute(self, path: str) -> bool:
+        """Check if path is absolute."""
+        return os.path.isabs(path)
+
+    def get_path(self, path: str) -> Path:
+        """Get a Path object for the given path."""
+        resolved = self._resolve_path(path)
+        # Since _resolve_path returns a string for sandbox filesystem,
+        # we need to create a Path object from it
+        return Path(resolved)