@magpiecloud/mags 1.5.1 → 1.6.0

package/python/INTEGRATION.md

@@ -0,0 +1,747 @@
+# Integrating Mags into Python Applications
+
+Mags gives your Python app on-demand Linux VMs that boot in ~300ms. Run untrusted code, build CI pipelines, host ephemeral services, or give your users their own sandboxed environments — all from a few lines of Python.
+
+```bash
+pip install magpie-mags
+```
+
+```python
+from mags import Mags
+
+m = Mags(api_token="your-token")
+result = m.run_and_wait("echo 'Hello from a VM!'")
+print(result["exit_code"])  # 0
+```
+
+---
+
+## Table of Contents
+
+- [Authentication](#authentication)
+- [Pattern 1: Code Execution Sandbox](#pattern-1-code-execution-sandbox)
+- [Pattern 2: AI Agent Tool Use](#pattern-2-ai-agent-tool-use)
+- [Pattern 3: Background Job Processing](#pattern-3-background-job-processing)
+- [Pattern 4: On-Demand Dev Environments](#pattern-4-on-demand-dev-environments)
+- [Pattern 5: CI/CD Pipeline Steps](#pattern-5-cicd-pipeline-steps)
+- [Pattern 6: Scheduled Tasks](#pattern-6-scheduled-tasks)
+- [Pattern 7: Web App with Live Preview](#pattern-7-web-app-with-live-preview)
+- [Working with Workspaces](#working-with-workspaces)
+- [File Uploads](#file-uploads)
+- [SSH Access](#ssh-access)
+- [Error Handling](#error-handling)
+- [Async / FastAPI Integration](#async--fastapi-integration)
+- [Django Integration](#django-integration)
+- [Flask Integration](#flask-integration)
+- [Environment & Configuration](#environment--configuration)
+
+---
+
+## Authentication
+
+Set your API token once. All calls use it automatically.
+
+```python
+# Option 1: Pass directly
+m = Mags(api_token="your-token")
+
+# Option 2: Environment variable (recommended for production)
+# export MAGS_API_TOKEN="your-token"
+m = Mags()
+
+# Option 3: Custom API endpoint (self-hosted)
+m = Mags(api_url="https://your-instance.example.com")
+```
+
+---
+
+## Pattern 1: Code Execution Sandbox
+
+Run user-submitted code safely inside an isolated VM. The code never touches your infrastructure.
+
+```python
+from mags import Mags, MagsError
+
+m = Mags()
+
+def execute_user_code(language, code, timeout=30):
+    """Run untrusted user code in a sandboxed VM."""
+
+    runners = {
+        "python": f"python3 -c {shlex.quote(code)}",
+        "javascript": f"node -e {shlex.quote(code)}",
+        "bash": code,
+    }
+
+    script = runners.get(language)
+    if not script:
+        return {"error": f"Unsupported language: {language}"}
+
+    try:
+        result = m.run_and_wait(script, timeout=timeout)
+        return {
+            "exit_code": result["exit_code"],
+            "output": [
+                log["message"]
+                for log in result["logs"]
+                if log["source"] in ("stdout", "stderr")
+            ],
+        }
+    except MagsError as e:
+        return {"error": str(e)}
+```
+
+```python
+# Usage
+import shlex
+
+output = execute_user_code("python", "print(sum(range(100)))")
+# {"exit_code": 0, "output": ["4950"]}
+```
+
+### With package installation
+
+```python
+def run_with_packages(code, packages):
+    """Run Python code with pip packages pre-installed."""
+    install = f"pip install -q {' '.join(packages)}" if packages else "true"
+    script = f"{install} && python3 -c {shlex.quote(code)}"
+    return m.run_and_wait(script, timeout=60)
+
+result = run_with_packages(
+    "import pandas as pd; print(pd.DataFrame({'a': [1,2,3]}))",
+    ["pandas"],
+)
+```
+
+### With a pre-built base image
+
+For repeated runs, avoid re-installing packages every time by using a base workspace:
+
+```python
+# One-time setup: create a base workspace with common packages
+m.run_and_wait(
+    "pip install pandas numpy requests flask scikit-learn",
+    workspace_id="python-base",
+)
+
+# Every subsequent run inherits the base (read-only, no install needed)
+result = m.run_and_wait(
+    "python3 -c 'import pandas; print(pandas.__version__)'",
+    base_workspace_id="python-base",
+)
+```
+
+---
+
+## Pattern 2: AI Agent Tool Use
+
+Give LLM agents a sandboxed environment to execute code, install packages, and inspect results.
+
+```python
+from mags import Mags
+
+m = Mags()
+
+def agent_tool_execute(code, workspace_id="agent-workspace"):
+    """Tool function for an AI agent to run code."""
+    result = m.run_and_wait(
+        code,
+        workspace_id=workspace_id,
+        timeout=60,
+    )
+
+    stdout = "\n".join(
+        log["message"]
+        for log in result["logs"]
+        if log["source"] == "stdout"
+    )
+    stderr = "\n".join(
+        log["message"]
+        for log in result["logs"]
+        if log["source"] == "stderr"
+    )
+
+    return {
+        "success": result["exit_code"] == 0,
+        "stdout": stdout,
+        "stderr": stderr,
+    }
+```
+
+### Claude / OpenAI tool definition
+
+```python
+tool_definition = {
+    "name": "execute_code",
+    "description": (
+        "Execute shell commands or code in a persistent Linux VM. "
+        "Installed packages and files persist between calls. "
+        "The VM runs Alpine Linux with Python, Node.js, and common tools."
+    ),
+    "input_schema": {
+        "type": "object",
+        "properties": {
+            "code": {
+                "type": "string",
+                "description": "Shell command(s) to execute",
+            }
+        },
+        "required": ["code"],
+    },
+}
+
+# In your agent loop:
+if tool_call.name == "execute_code":
+    result = agent_tool_execute(tool_call.input["code"])
+    # Return result to the LLM
+```
+
+### Multi-turn agent with persistent workspace
+
+```python
+class AgentSandbox:
+    """Per-session sandbox that persists state across tool calls."""
+
+    def __init__(self, session_id):
+        self.m = Mags()
+        self.workspace_id = f"agent-{session_id}"
+
+    def execute(self, code):
+        return self.m.run_and_wait(
+            code,
+            workspace_id=self.workspace_id,
+            timeout=60,
+        )
+
+    def upload_and_run(self, file_path, command):
+        file_ids = self.m.upload_files([file_path])
+        return self.m.run_and_wait(
+            command,
+            workspace_id=self.workspace_id,
+            file_ids=file_ids,
+            timeout=60,
+        )
+
+# Each user conversation gets its own sandbox
+sandbox = AgentSandbox(session_id="user-123-conv-456")
+sandbox.execute("pip install matplotlib")
+sandbox.execute("python3 -c 'import matplotlib; print(matplotlib.__version__)'")
+# matplotlib is still installed on the next call — workspace persists
+```
+
+---
+
+## Pattern 3: Background Job Processing
+
+Offload heavy or untrusted processing to VMs. Good for video transcoding, PDF generation, data pipelines, etc.
+
+```python
+import time
+from mags import Mags
+
+m = Mags()
+
+def process_video(video_url, output_format="mp4"):
+    """Transcode a video in a sandboxed VM."""
+    script = f"""
+apk add ffmpeg
+curl -sL '{video_url}' -o /tmp/input
+ffmpeg -i /tmp/input -c:v libx264 /tmp/output.{output_format}
+ls -la /tmp/output.*
+"""
+    return m.run_and_wait(script, timeout=300)
+
+
+def run_data_pipeline(sql_query, workspace_id="etl-pipeline"):
+    """Run a data processing pipeline with persistent deps."""
+    script = f"""
+python3 << 'PYEOF'
+import sqlite3, json
+
+conn = sqlite3.connect("/workspace/data.db")
+cursor = conn.execute("{sql_query}")
+rows = cursor.fetchall()
+print(json.dumps(rows))
+PYEOF
+"""
+    return m.run_and_wait(script, workspace_id=workspace_id, timeout=120)
+```
+
+### Fire-and-forget (don't block)
+
+```python
+def submit_job(script, name=None):
+    """Submit a job without waiting. Check status later."""
+    result = m.run(script, name=name)
+    return result["request_id"]
+
+def check_job(request_id):
+    """Poll for completion."""
+    status = m.status(request_id)
+    if status["status"] in ("completed", "error"):
+        logs = m.logs(request_id)
+        return {**status, "logs": logs.get("logs", [])}
+    return status
+
+# Submit
+job_id = submit_job("sleep 30 && echo done", name="background-task")
+
+# Check later
+result = check_job(job_id)
+```
+
+---
+
+## Pattern 4: On-Demand Dev Environments
+
+Give each user a persistent sandbox with SSH and URL access.
+
+```python
+from mags import Mags
+
+m = Mags()
+
+def create_dev_environment(user_id, template="python"):
+    """Spin up a dev environment for a user."""
+    templates = {
+        "python": "apk add python3 py3-pip git && pip install ipython",
+        "node": "apk add nodejs npm git && npm i -g yarn",
+        "go": "apk add go git",
+    }
+
+    workspace_id = f"dev-{user_id}"
+    setup_script = templates.get(template, templates["python"])
+
+    # Create persistent VM with the dev environment
+    job = m.run(
+        setup_script,
+        workspace_id=workspace_id,
+        persistent=True,
+        startup_command=templates[template].split("&&")[-1].strip(),
+    )
+    request_id = job["request_id"]
+
+    # Wait for it to be ready
+    import time
+    for _ in range(30):
+        status = m.status(request_id)
+        if status["status"] == "running":
+            break
+        time.sleep(1)
+
+    # Enable SSH access
+    ssh = m.enable_access(request_id, port=22)
+
+    return {
+        "request_id": request_id,
+        "workspace_id": workspace_id,
+        "ssh_host": ssh["ssh_host"],
+        "ssh_port": ssh["ssh_port"],
+        "ssh_command": (
+            f"ssh -p {ssh['ssh_port']} "
+            f"-o StrictHostKeyChecking=no "
+            f"root@{ssh['ssh_host']}"
+        ),
+    }
+```
+
+---
+
+## Pattern 5: CI/CD Pipeline Steps
+
+Run test suites, builds, or linting in isolated VMs.
+
+```python
+from mags import Mags
+
+m = Mags()
+
+def run_tests(repo_url, branch="main", workspace_id=None):
+    """Clone a repo and run its test suite."""
+    script = f"""
+apk add git nodejs npm
+git clone --branch {branch} --depth 1 {repo_url} /tmp/repo
+cd /tmp/repo
+npm ci
+npm test
+"""
+    result = m.run_and_wait(script, workspace_id=workspace_id, timeout=300)
+    return {
+        "passed": result["exit_code"] == 0,
+        "duration_ms": result["duration_ms"],
+        "output": [l["message"] for l in result["logs"] if l["source"] == "stdout"],
+    }
+
+
+def parallel_test_matrix(repo_url, versions):
+    """Run tests against multiple Node.js versions in parallel."""
+    jobs = {}
+    for version in versions:
+        script = f"""
+apk add nodejs~={version} npm git
+git clone --depth 1 {repo_url} /tmp/repo
+cd /tmp/repo && npm ci && npm test
+"""
+        result = m.run(script, name=f"test-node-{version}")
+        jobs[version] = result["request_id"]
+
+    # Poll all jobs
+    import time
+    results = {}
+    pending = set(jobs.keys())
+
+    while pending:
+        for version in list(pending):
+            status = m.status(jobs[version])
+            if status["status"] in ("completed", "error"):
+                results[version] = {
+                    "passed": status.get("exit_code") == 0,
+                    "status": status["status"],
+                }
+                pending.discard(version)
+        if pending:
+            time.sleep(2)
+
+    return results
+
+# results = parallel_test_matrix("https://github.com/user/repo.git", ["18", "20", "22"])
+```
+
+---
+
+## Pattern 6: Scheduled Tasks
+
+Use Mags cron jobs for recurring work.
+
+```python
+from mags import Mags
+
+m = Mags()
+
+# Run a health check every 5 minutes
+cron = m.cron_create(
+    name="health-check",
+    cron_expression="*/5 * * * *",
+    script='curl -sf https://myapp.com/health || echo "ALERT: health check failed"',
+)
+
+# Nightly database backup
+m.cron_create(
+    name="db-backup",
+    cron_expression="0 2 * * *",
+    script="pg_dump $DATABASE_URL | gzip > /workspace/backup-$(date +%F).sql.gz",
+    workspace_id="backups",
+)
+
+# List all cron jobs
+crons = m.cron_list()
+for job in crons.get("cron_jobs", []):
+    print(f"{job['name']}: {job['cron_expression']} (enabled={job['enabled']})")
+
+# Pause a cron job
+m.cron_update(cron["id"], enabled=False)
+
+# Delete it
+m.cron_delete(cron["id"])
+```
+
+---
+
+## Pattern 7: Web App with Live Preview
+
+Deploy user code and give them a live URL.
+
+```python
+from mags import Mags
+
+m = Mags()
+
+def deploy_preview(user_id, html_content):
+    """Deploy HTML and return a live preview URL."""
+    import shlex
+
+    script = f"""
+mkdir -p /workspace/site
+cat > /workspace/site/index.html << 'HTMLEOF'
+{html_content}
+HTMLEOF
+cd /workspace/site && python3 -m http.server 8080
+"""
+    job = m.run(
+        script,
+        workspace_id=f"preview-{user_id}",
+        persistent=True,
+        startup_command="cd /workspace/site && python3 -m http.server 8080",
+    )
+
+    # Wait for VM to start
+    import time
+    for _ in range(15):
+        status = m.status(job["request_id"])
+        if status["status"] == "running":
+            break
+        time.sleep(1)
+
+    # Enable URL access
+    access = m.enable_access(job["request_id"], port=8080)
+    return {
+        "request_id": job["request_id"],
+        "url": access.get("url"),
+        "status": "live",
+    }
+
+# preview = deploy_preview("user-42", "<h1>My App</h1>")
+# print(preview["url"])  # https://abc123.apps.magpiecloud.com
+```
+
+The URL stays live. When the VM goes idle it sleeps automatically. The next visitor triggers an auto-wake (~3-5 seconds) and the page loads.
+
+---
+
+## Working with Workspaces
+
+Workspaces are persistent filesystems backed by S3. Files, installed packages, and configs survive across VM restarts and sleep/wake cycles.
+
+```python
+# Create and populate a workspace
+m.run_and_wait("pip install flask gunicorn", workspace_id="my-app")
+
+# Run again — flask is already installed
+m.run_and_wait("flask --version", workspace_id="my-app")
+
+# Fork a workspace: start from base, save changes to a new workspace
+m.run_and_wait(
+    "pip install pandas",
+    workspace_id="my-app-with-pandas",
+    base_workspace_id="my-app",
+)
+
+# Ephemeral run: no workspace sync, fastest possible
+m.run_and_wait("echo 'no persistence'", ephemeral=True)
+```
+
+| Mode | `workspace_id` | `base_workspace_id` | Behavior |
+|------|----------------|----------------------|----------|
+| Ephemeral | omit | omit | No persistence. Fastest. |
+| Persistent | `"my-ws"` | omit | Read-write. Changes sync to S3. |
+| Read-only base | omit | `"my-base"` | Base mounted read-only. Changes discarded. |
+| Fork | `"fork-1"` | `"my-base"` | Starts from base, saves to `fork-1`. |
+
+---
+
+## File Uploads
+
+Upload local files into a VM before the script runs. Files appear in `/root/`.
+
+```python
+# Upload and use files
+file_ids = m.upload_files(["model.pkl", "data.csv"])
+
+result = m.run_and_wait(
+    "python3 -c 'import pickle; m = pickle.load(open(\"/root/model.pkl\",\"rb\")); print(type(m))'",
+    file_ids=file_ids,
+    timeout=30,
+)
+```
+
+---
+
+## SSH Access
+
+Enable SSH to get a full interactive terminal or run remote commands programmatically.
+
+```python
+import subprocess, tempfile, os
+
+def ssh_command(request_id, command=None):
+    """Run a command via SSH on a running VM."""
+    access = m.enable_access(request_id, port=22)
+
+    # Write private key to temp file
+    key_file = tempfile.NamedTemporaryFile(mode="w", suffix=".pem", delete=False)
+    key_file.write(access["ssh_private_key"])
+    key_file.close()
+    os.chmod(key_file.name, 0o600)
+
+    ssh_cmd = [
+        "ssh", "-i", key_file.name,
+        "-p", str(access["ssh_port"]),
+        "-o", "StrictHostKeyChecking=no",
+        "-o", "UserKnownHostsFile=/dev/null",
+        f"root@{access['ssh_host']}",
+    ]
+
+    if command:
+        ssh_cmd.append(command)
+        result = subprocess.run(ssh_cmd, capture_output=True, text=True, timeout=30)
+        os.unlink(key_file.name)
+        return {"stdout": result.stdout, "stderr": result.stderr, "returncode": result.returncode}
+    else:
+        # Interactive — opens a terminal
+        os.unlink(key_file.name)
+        return ssh_cmd  # caller can use subprocess.run(ssh_cmd)
+```
+
+---
+
+## Error Handling
+
+```python
+from mags import Mags, MagsError
+
+m = Mags()
+
+try:
+    result = m.run_and_wait("exit 1", timeout=10)
+    if result["exit_code"] != 0:
+        print("Script failed:", result["exit_code"])
+        for log in result["logs"]:
+            if log["source"] == "stderr":
+                print("  ", log["message"])
+
+except MagsError as e:
+    # API-level errors (auth, not found, timeout, etc.)
+    print(f"Mags error: {e}")
+    if e.status_code == 401:
+        print("Check your API token")
+```
+
+---
+
+## Async / FastAPI Integration
+
+The SDK uses synchronous `requests`. For async frameworks, run it in a thread pool:
+
+```python
+import asyncio
+from functools import partial
+
+from fastapi import FastAPI
+from mags import Mags
+
+app = FastAPI()
+m = Mags()
+
+@app.post("/execute")
+async def execute(code: str, language: str = "python"):
+    runner = f"python3 -c {code!r}" if language == "python" else code
+
+    # Run synchronous SDK in a thread to avoid blocking the event loop
+    loop = asyncio.get_event_loop()
+    result = await loop.run_in_executor(
+        None,
+        partial(m.run_and_wait, runner, timeout=30),
+    )
+
+    return {
+        "exit_code": result["exit_code"],
+        "output": [l["message"] for l in result["logs"] if l["source"] == "stdout"],
+    }
+```
+
+---
+
+## Django Integration
+
+```python
+# settings.py
+MAGS_API_TOKEN = os.environ.get("MAGS_API_TOKEN")
+
+# services.py
+from mags import Mags
+from django.conf import settings
+
+_client = None
+
+def get_mags_client():
+    global _client
+    if _client is None:
+        _client = Mags(api_token=settings.MAGS_API_TOKEN)
+    return _client
+
+# views.py
+from django.http import JsonResponse
+from .services import get_mags_client
+
+def run_code(request):
+    m = get_mags_client()
+    code = request.POST.get("code", "echo hello")
+
+    result = m.run_and_wait(code, timeout=30)
+
+    return JsonResponse({
+        "exit_code": result["exit_code"],
+        "output": [l["message"] for l in result["logs"] if l["source"] == "stdout"],
+    })
+```
+
+---
+
+## Flask Integration
+
+```python
+from flask import Flask, request, jsonify
+from mags import Mags
+
+app = Flask(__name__)
+m = Mags()
+
+@app.post("/run")
+def run():
+    data = request.get_json()
+    result = m.run_and_wait(
+        data["script"],
+        workspace_id=data.get("workspace_id"),
+        timeout=data.get("timeout", 30),
+    )
+    return jsonify({
+        "exit_code": result["exit_code"],
+        "logs": result["logs"],
+    })
+```
+
+---
+
+## Environment & Configuration
+
+| Env Variable | Description | Default |
+|-------------|-------------|---------|
+| `MAGS_API_TOKEN` | API token (required) | — |
+| `MAGS_TOKEN` | Alias for `MAGS_API_TOKEN` | — |
+| `MAGS_API_URL` | API base URL | `https://api.magpiecloud.com` |
+
+```python
+# All configuration options
+m = Mags(
+    api_token="...",                          # required
+    api_url="https://api.magpiecloud.com",    # optional
+    timeout=30,                               # default request timeout in seconds
+)
+```
+
+---
+
+## VM Specs
+
+| Property | Value |
+|----------|-------|
+| OS | Alpine Linux |
+| Shell | `/bin/sh` (ash) |
+| Package manager | `apk add <package>` |
+| User | `root` |
+| Working directory | `/root` |
+| Boot time | ~300ms from pool |
+| Default timeout | 300 seconds |
+
+Common packages: `python3`, `py3-pip`, `nodejs`, `npm`, `git`, `curl`, `jq`, `ffmpeg`, `go`, `rust`, `gcc`.
+
+---
+
+## Links
+
+- **Website:** [mags.run](https://mags.run)
+- **PyPI:** `pip install magpie-mags`
+- **npm:** `npm install @magpiecloud/mags`
+- **API Reference:** [API.md](../API.md)
+- **CLI Quickstart:** [QUICKSTART.md](../QUICKSTART.md)