@magpiecloud/mags 1.8.13 → 1.8.15

package/python/INTEGRATION.md

@@ -1,800 +0,0 @@
-# Integrating Mags into Python Applications
-
-Mags gives your Python app on-demand Linux VMs that boot in ~300ms. Run untrusted code, build CI pipelines, host ephemeral services, or give your users their own sandboxed environments — all from a few lines of Python.
-
-```bash
-pip install magpie-mags
-```
-
-```python
-from mags import Mags
-
-m = Mags(api_token="your-token")
-result = m.run_and_wait("echo 'Hello from a VM!'")
-print(result["exit_code"])  # 0
-```
-
----
-
-## Table of Contents
-
-- [Authentication](#authentication)
-- [Pattern 1: Code Execution Sandbox](#pattern-1-code-execution-sandbox)
-- [Pattern 2: AI Agent Tool Use](#pattern-2-ai-agent-tool-use)
-- [Pattern 3: Background Job Processing](#pattern-3-background-job-processing)
-- [Pattern 4: On-Demand Dev Environments](#pattern-4-on-demand-dev-environments)
-- [Pattern 5: CI/CD Pipeline Steps](#pattern-5-cicd-pipeline-steps)
-- [Pattern 6: Scheduled Tasks](#pattern-6-scheduled-tasks)
-- [Pattern 7: Web App with Live Preview](#pattern-7-web-app-with-live-preview)
-- [Working with Workspaces](#working-with-workspaces)
-- [File Uploads](#file-uploads)
-- [SSH Access](#ssh-access)
-- [Error Handling](#error-handling)
-- [Async / FastAPI Integration](#async--fastapi-integration)
-- [Django Integration](#django-integration)
-- [Flask Integration](#flask-integration)
-- [Environment & Configuration](#environment--configuration)
-
----
-
-## Authentication
-
-Set your API token once. All calls use it automatically.
-
-```python
-# Option 1: Pass directly
-m = Mags(api_token="your-token")
-
-# Option 2: Environment variable (recommended for production)
-# export MAGS_API_TOKEN="your-token"
-m = Mags()
-
-# Option 3: Custom API endpoint (self-hosted)
-m = Mags(api_url="https://your-instance.example.com")
-```
-
----
-
-## Pattern 1: Code Execution Sandbox
-
-Run user-submitted code safely inside an isolated VM. The code never touches your infrastructure.
-
-```python
-from mags import Mags, MagsError
-
-m = Mags()
-
-def execute_user_code(language, code, timeout=30):
-    """Run untrusted user code in a sandboxed VM."""
-
-    runners = {
-        "python": f"python3 -c {shlex.quote(code)}",
-        "javascript": f"node -e {shlex.quote(code)}",
-        "bash": code,
-    }
-
-    script = runners.get(language)
-    if not script:
-        return {"error": f"Unsupported language: {language}"}
-
-    try:
-        result = m.run_and_wait(script, timeout=timeout)
-        return {
-            "exit_code": result["exit_code"],
-            "output": [
-                log["message"]
-                for log in result["logs"]
-                if log["source"] in ("stdout", "stderr")
-            ],
-        }
-    except MagsError as e:
-        return {"error": str(e)}
-```
-
-```python
-# Usage
-import shlex
-
-output = execute_user_code("python", "print(sum(range(100)))")
-# {"exit_code": 0, "output": ["4950"]}
-```
-
-### With package installation
-
-```python
-def run_with_packages(code, packages):
-    """Run Python code with pip packages pre-installed."""
-    install = f"pip install -q {' '.join(packages)}" if packages else "true"
-    script = f"{install} && python3 -c {shlex.quote(code)}"
-    return m.run_and_wait(script, timeout=60)
-
-result = run_with_packages(
-    "import pandas as pd; print(pd.DataFrame({'a': [1,2,3]}))",
-    ["pandas"],
-)
-```
-
-### With a pre-built base image
-
-For repeated runs, avoid re-installing packages every time by creating a base workspace and syncing it:
-
-```python
-# One-time setup: create a base workspace with common packages
-job = m.run(
-    "pip install pandas numpy requests flask scikit-learn",
-    workspace_id="python-base",
-    persistent=True,
-)
-
-# Wait for setup to finish, then sync to S3
-import time
-for _ in range(60):
-    status = m.status(job["request_id"])
-    if status["status"] == "running":
-        break
-    time.sleep(1)
-
-# Force sync — persists everything to S3 immediately
-m.sync(job["request_id"])
-
-# Every subsequent run inherits the base (read-only, no install needed)
-result = m.run_and_wait(
-    "python3 -c 'import pandas; print(pandas.__version__)'",
-    base_workspace_id="python-base",
-)
-
-# Fork: load base, save changes to a new workspace
-result = m.run_and_wait(
-    "pip install torch",
-    base_workspace_id="python-base",
-    workspace_id="python-ml",
-)
-```
-
----
-
-## Pattern 2: AI Agent Tool Use
-
-Give LLM agents a sandboxed environment to execute code, install packages, and inspect results.
-
-```python
-from mags import Mags
-
-m = Mags()
-
-def agent_tool_execute(code, workspace_id="agent-workspace"):
-    """Tool function for an AI agent to run code."""
-    result = m.run_and_wait(
-        code,
-        workspace_id=workspace_id,
-        timeout=60,
-    )
-
-    stdout = "\n".join(
-        log["message"]
-        for log in result["logs"]
-        if log["source"] == "stdout"
-    )
-    stderr = "\n".join(
-        log["message"]
-        for log in result["logs"]
-        if log["source"] == "stderr"
-    )
-
-    return {
-        "success": result["exit_code"] == 0,
-        "stdout": stdout,
-        "stderr": stderr,
-    }
-```
-
-### Claude / OpenAI tool definition
-
-```python
-tool_definition = {
-    "name": "execute_code",
-    "description": (
-        "Execute shell commands or code in a persistent Linux VM. "
-        "Installed packages and files persist between calls. "
-        "The VM runs Alpine Linux with Python, Node.js, and common tools."
-    ),
-    "input_schema": {
-        "type": "object",
-        "properties": {
-            "code": {
-                "type": "string",
-                "description": "Shell command(s) to execute",
-            }
-        },
-        "required": ["code"],
-    },
-}
-
-# In your agent loop:
-if tool_call.name == "execute_code":
-    result = agent_tool_execute(tool_call.input["code"])
-    # Return result to the LLM
-```
-
-### Multi-turn agent with persistent workspace
-
-```python
-class AgentSandbox:
-    """Per-session sandbox that persists state across tool calls."""
-
-    def __init__(self, session_id):
-        self.m = Mags()
-        self.workspace_id = f"agent-{session_id}"
-
-    def execute(self, code):
-        return self.m.run_and_wait(
-            code,
-            workspace_id=self.workspace_id,
-            timeout=60,
-        )
-
-    def upload_and_run(self, file_path, command):
-        file_ids = self.m.upload_files([file_path])
-        return self.m.run_and_wait(
-            command,
-            workspace_id=self.workspace_id,
-            file_ids=file_ids,
-            timeout=60,
-        )
-
-# Each user conversation gets its own sandbox
-sandbox = AgentSandbox(session_id="user-123-conv-456")
-sandbox.execute("pip install matplotlib")
-sandbox.execute("python3 -c 'import matplotlib; print(matplotlib.__version__)'")
-# matplotlib is still installed on the next call — workspace persists
-```
-
----
-
-## Pattern 3: Background Job Processing
-
-Offload heavy or untrusted processing to VMs. Good for video transcoding, PDF generation, data pipelines, etc.
-
-```python
-import time
-from mags import Mags
-
-m = Mags()
-
-def process_video(video_url, output_format="mp4"):
-    """Transcode a video in a sandboxed VM."""
-    script = f"""
-apk add ffmpeg
-curl -sL '{video_url}' -o /tmp/input
-ffmpeg -i /tmp/input -c:v libx264 /tmp/output.{output_format}
-ls -la /tmp/output.*
-"""
-    return m.run_and_wait(script, timeout=300)
-
-
-def run_data_pipeline(sql_query, workspace_id="etl-pipeline"):
-    """Run a data processing pipeline with persistent deps."""
-    script = f"""
-python3 << 'PYEOF'
-import sqlite3, json
-
-conn = sqlite3.connect("/root/data.db")
-cursor = conn.execute("{sql_query}")
-rows = cursor.fetchall()
-print(json.dumps(rows))
-PYEOF
-"""
-    return m.run_and_wait(script, workspace_id=workspace_id, timeout=120)
-```
-
-### Fire-and-forget (don't block)
-
-```python
-def submit_job(script, name=None):
-    """Submit a job without waiting. Check status later."""
-    result = m.run(script, name=name)
-    return result["request_id"]
-
-def check_job(request_id):
-    """Poll for completion."""
-    status = m.status(request_id)
-    if status["status"] in ("completed", "error"):
-        logs = m.logs(request_id)
-        return {**status, "logs": logs.get("logs", [])}
-    return status
-
-# Submit
-job_id = submit_job("sleep 30 && echo done", name="background-task")
-
-# Check later
-result = check_job(job_id)
-```
-
----
-
-## Pattern 4: On-Demand Dev Environments
-
-Give each user a persistent sandbox with SSH and URL access.
-
-```python
-from mags import Mags
-
-m = Mags()
-
-def create_dev_environment(user_id, template="python"):
-    """Spin up a dev environment for a user."""
-    templates = {
-        "python": "apk add python3 py3-pip git && pip install ipython",
-        "node": "apk add nodejs npm git && npm i -g yarn",
-        "go": "apk add go git",
-    }
-
-    workspace_id = f"dev-{user_id}"
-    setup_script = templates.get(template, templates["python"])
-
-    # Create persistent VM with the dev environment
-    job = m.run(
-        setup_script,
-        workspace_id=workspace_id,
-        persistent=True,
-        startup_command=templates[template].split("&&")[-1].strip(),
-    )
-    request_id = job["request_id"]
-
-    # Wait for it to be ready
-    import time
-    for _ in range(30):
-        status = m.status(request_id)
-        if status["status"] == "running":
-            break
-        time.sleep(1)
-
-    # Enable SSH access
-    ssh = m.enable_access(request_id, port=22)
-
-    return {
-        "request_id": request_id,
-        "workspace_id": workspace_id,
-        "ssh_host": ssh["ssh_host"],
-        "ssh_port": ssh["ssh_port"],
-        "ssh_command": (
-            f"ssh -p {ssh['ssh_port']} "
-            f"-o StrictHostKeyChecking=no "
-            f"root@{ssh['ssh_host']}"
-        ),
-    }
-```
-
----
-
-## Pattern 5: CI/CD Pipeline Steps
-
-Run test suites, builds, or linting in isolated VMs.
-
-```python
-from mags import Mags
-
-m = Mags()
-
-def run_tests(repo_url, branch="main", workspace_id=None):
-    """Clone a repo and run its test suite."""
-    script = f"""
-apk add git nodejs npm
-git clone --branch {branch} --depth 1 {repo_url} /tmp/repo
-cd /tmp/repo
-npm ci
-npm test
-"""
-    result = m.run_and_wait(script, workspace_id=workspace_id, timeout=300)
-    return {
-        "passed": result["exit_code"] == 0,
-        "duration_ms": result["duration_ms"],
-        "output": [l["message"] for l in result["logs"] if l["source"] == "stdout"],
-    }
-
-
-def parallel_test_matrix(repo_url, versions):
-    """Run tests against multiple Node.js versions in parallel."""
-    jobs = {}
-    for version in versions:
-        script = f"""
-apk add nodejs~={version} npm git
-git clone --depth 1 {repo_url} /tmp/repo
-cd /tmp/repo && npm ci && npm test
-"""
-        result = m.run(script, name=f"test-node-{version}")
-        jobs[version] = result["request_id"]
-
-    # Poll all jobs
-    import time
-    results = {}
-    pending = set(jobs.keys())
-
-    while pending:
-        for version in list(pending):
-            status = m.status(jobs[version])
-            if status["status"] in ("completed", "error"):
-                results[version] = {
-                    "passed": status.get("exit_code") == 0,
-                    "status": status["status"],
-                }
-                pending.discard(version)
-        if pending:
-            time.sleep(2)
-
-    return results
-
-# results = parallel_test_matrix("https://github.com/user/repo.git", ["18", "20", "22"])
-```
-
----
-
-## Pattern 6: Scheduled Tasks
-
-Use Mags cron jobs for recurring work.
-
-```python
-from mags import Mags
-
-m = Mags()
-
-# Run a health check every 5 minutes
-cron = m.cron_create(
-    name="health-check",
-    cron_expression="*/5 * * * *",
-    script='curl -sf https://myapp.com/health || echo "ALERT: health check failed"',
-)
-
-# Nightly database backup
-m.cron_create(
-    name="db-backup",
-    cron_expression="0 2 * * *",
-    script="pg_dump $DATABASE_URL | gzip > /root/backup-$(date +%F).sql.gz",
-    workspace_id="backups",
-)
-
-# List all cron jobs
-crons = m.cron_list()
-for job in crons.get("cron_jobs", []):
-    print(f"{job['name']}: {job['cron_expression']} (enabled={job['enabled']})")
-
-# Pause a cron job
-m.cron_update(cron["id"], enabled=False)
-
-# Delete it
-m.cron_delete(cron["id"])
-```
-
----
-
-## Pattern 7: Web App with Live Preview
-
-Deploy user code and give them a live URL.
-
-```python
-from mags import Mags
-
-m = Mags()
-
-def deploy_preview(user_id, html_content):
-    """Deploy HTML and return a live preview URL."""
-    import shlex
-
-    script = f"""
-mkdir -p /root/site
-cat > /root/site/index.html << 'HTMLEOF'
-{html_content}
-HTMLEOF
-cd /root/site && python3 -m http.server 8080
-"""
-    job = m.run(
-        script,
-        workspace_id=f"preview-{user_id}",
-        persistent=True,
-        startup_command="cd /root/site && python3 -m http.server 8080",
-    )
-
-    # Wait for VM to start
-    import time
-    for _ in range(15):
-        status = m.status(job["request_id"])
-        if status["status"] == "running":
-            break
-        time.sleep(1)
-
-    # Enable URL access
-    access = m.enable_access(job["request_id"], port=8080)
-    return {
-        "request_id": job["request_id"],
-        "url": access.get("url"),
-        "status": "live",
-    }
-
-# preview = deploy_preview("user-42", "<h1>My App</h1>")
-# print(preview["url"])  # https://abc123.apps.magpiecloud.com
-```
-
-The URL stays live. When the VM goes idle it sleeps automatically. The next visitor triggers an auto-wake (~3-5 seconds) and the page loads.
-
----
-
-## Working with Workspaces
-
-Workspaces are persistent filesystems backed by S3. Files, installed packages, and configs survive across VM restarts and sleep/wake cycles.
-
-```python
-# Create and populate a workspace
-m.run_and_wait("pip install flask gunicorn", workspace_id="my-app")
-
-# Run again — flask is already installed
-m.run_and_wait("flask --version", workspace_id="my-app")
-
-# Fork a workspace: start from base, save changes to a new workspace
-m.run_and_wait(
-    "pip install pandas",
-    workspace_id="my-app-with-pandas",
-    base_workspace_id="my-app",
-)
-
-# Ephemeral run: no workspace sync, fastest possible
-m.run_and_wait("echo 'no persistence'", ephemeral=True)
-```
-
-| Mode | `workspace_id` | `base_workspace_id` | Behavior |
-|------|----------------|----------------------|----------|
-| Ephemeral | omit | omit | No persistence. Fastest. |
-| Persistent | `"my-ws"` | omit | Read-write. Changes sync to S3. |
-| Read-only base | omit | `"my-base"` | Base mounted read-only. Changes discarded. |
-| Fork | `"fork-1"` | `"my-base"` | Starts from base, saves to `fork-1`. |
-
-### Syncing workspaces
-
-Workspaces sync to S3 automatically when a job completes. For persistent VMs (`persistent=True`), workspaces also sync every 30 seconds and on sleep.
-
-Use `m.sync()` to force an immediate sync without stopping the VM — useful for persisting a base image you've just set up:
-
-```python
-# Set up a base workspace on a persistent VM
-job = m.run(
-    "pip install pandas numpy scikit-learn",
-    workspace_id="ml-base",
-    persistent=True,
-)
-
-# Wait for the install to finish
-import time
-for _ in range(60):
-    status = m.status(job["request_id"])
-    if status["status"] == "running":
-        break
-    time.sleep(1)
-
-# Force sync — base image is now available for other jobs
-m.sync(job["request_id"])
-
-# List and manage workspaces
-workspaces = m.list_workspaces()
-for ws in workspaces.get("workspaces", []):
-    print(f"{ws['workspace_id']} — {ws['job_count']} jobs")
-
-# Delete a workspace (removes stored data from S3)
-m.delete_workspace("old-workspace")
-```
-
----
-
-## File Uploads
-
-Upload local files into a VM before the script runs. Files appear in `/root/`.
-
-```python
-# Upload and use files
-file_ids = m.upload_files(["model.pkl", "data.csv"])
-
-result = m.run_and_wait(
-    "python3 -c 'import pickle; m = pickle.load(open(\"/root/model.pkl\",\"rb\")); print(type(m))'",
-    file_ids=file_ids,
-    timeout=30,
-)
-```
-
----
-
-## SSH Access
-
-Enable SSH to get a full interactive terminal or run remote commands programmatically.
-
-```python
-import subprocess, tempfile, os
-
-def ssh_command(request_id, command=None):
-    """Run a command via SSH on a running VM."""
-    access = m.enable_access(request_id, port=22)
-
-    # Write private key to temp file
-    key_file = tempfile.NamedTemporaryFile(mode="w", suffix=".pem", delete=False)
-    key_file.write(access["ssh_private_key"])
-    key_file.close()
-    os.chmod(key_file.name, 0o600)
-
-    ssh_cmd = [
-        "ssh", "-i", key_file.name,
-        "-p", str(access["ssh_port"]),
-        "-o", "StrictHostKeyChecking=no",
-        "-o", "UserKnownHostsFile=/dev/null",
-        f"root@{access['ssh_host']}",
-    ]
-
-    if command:
-        ssh_cmd.append(command)
-        result = subprocess.run(ssh_cmd, capture_output=True, text=True, timeout=30)
-        os.unlink(key_file.name)
-        return {"stdout": result.stdout, "stderr": result.stderr, "returncode": result.returncode}
-    else:
-        # Interactive — opens a terminal
-        os.unlink(key_file.name)
-        return ssh_cmd  # caller can use subprocess.run(ssh_cmd)
-```
-
----
-
-## Error Handling
-
-```python
-from mags import Mags, MagsError
-
-m = Mags()
-
-try:
-    result = m.run_and_wait("exit 1", timeout=10)
-    if result["exit_code"] != 0:
-        print("Script failed:", result["exit_code"])
-        for log in result["logs"]:
-            if log["source"] == "stderr":
-                print("  ", log["message"])
-
-except MagsError as e:
-    # API-level errors (auth, not found, timeout, etc.)
-    print(f"Mags error: {e}")
-    if e.status_code == 401:
-        print("Check your API token")
-```
-
----
-
-## Async / FastAPI Integration
-
-The SDK uses synchronous `requests`. For async frameworks, run it in a thread pool:
-
-```python
-import asyncio
-from functools import partial
-
-from fastapi import FastAPI
-from mags import Mags
-
-app = FastAPI()
-m = Mags()
-
-@app.post("/execute")
-async def execute(code: str, language: str = "python"):
-    runner = f"python3 -c {code!r}" if language == "python" else code
-
-    # Run synchronous SDK in a thread to avoid blocking the event loop
-    loop = asyncio.get_event_loop()
-    result = await loop.run_in_executor(
-        None,
-        partial(m.run_and_wait, runner, timeout=30),
-    )
-
-    return {
-        "exit_code": result["exit_code"],
-        "output": [l["message"] for l in result["logs"] if l["source"] == "stdout"],
-    }
-```
-
----
-
-## Django Integration
-
-```python
-# settings.py
-MAGS_API_TOKEN = os.environ.get("MAGS_API_TOKEN")
-
-# services.py
-from mags import Mags
-from django.conf import settings
-
-_client = None
-
-def get_mags_client():
-    global _client
-    if _client is None:
-        _client = Mags(api_token=settings.MAGS_API_TOKEN)
-    return _client
-
-# views.py
-from django.http import JsonResponse
-from .services import get_mags_client
-
-def run_code(request):
-    m = get_mags_client()
-    code = request.POST.get("code", "echo hello")
-
-    result = m.run_and_wait(code, timeout=30)
-
-    return JsonResponse({
-        "exit_code": result["exit_code"],
-        "output": [l["message"] for l in result["logs"] if l["source"] == "stdout"],
-    })
-```
-
----
-
-## Flask Integration
-
-```python
-from flask import Flask, request, jsonify
-from mags import Mags
-
-app = Flask(__name__)
-m = Mags()
-
-@app.post("/run")
-def run():
-    data = request.get_json()
-    result = m.run_and_wait(
-        data["script"],
-        workspace_id=data.get("workspace_id"),
-        timeout=data.get("timeout", 30),
-    )
-    return jsonify({
-        "exit_code": result["exit_code"],
-        "logs": result["logs"],
-    })
-```
-
----
-
-## Environment & Configuration
-
-| Env Variable | Description | Default |
-|-------------|-------------|---------|
-| `MAGS_API_TOKEN` | API token (required) | — |
-| `MAGS_TOKEN` | Alias for `MAGS_API_TOKEN` | — |
-| `MAGS_API_URL` | API base URL | `https://api.magpiecloud.com` |
-
-```python
-# All configuration options
-m = Mags(
-    api_token="...",                          # required
-    api_url="https://api.magpiecloud.com",    # optional
-    timeout=30,                               # default request timeout in seconds
-)
-```
-
----
-
-## VM Specs
-
-| Property | Value |
-|----------|-------|
-| OS | Alpine Linux |
-| Shell | `/bin/sh` (ash) |
-| Package manager | `apk add <package>` |
-| User | `root` |
-| Working directory | `/root` |
-| Boot time | ~300ms from pool |
-| Default timeout | 300 seconds |
-
-Common packages: `python3`, `py3-pip`, `nodejs`, `npm`, `git`, `curl`, `jq`, `ffmpeg`, `go`, `rust`, `gcc`.
-
----
-
-## Links
-
-- **Website:** [mags.run](https://mags.run)
-- **PyPI:** `pip install magpie-mags`
-- **npm:** `npm install @magpiecloud/mags`
-- **API Reference:** [API.md](../API.md)
-- **CLI Quickstart:** [QUICKSTART.md](../QUICKSTART.md)