@magpiecloud/mags 1.8.16 → 1.8.17

package/README.md

@@ -1,6 +1,17 @@
-# @magpiecloud/mags
+# Mags - Instant VM Execution
 
-Execute scripts instantly on Magpie's microVM infrastructure. VMs boot in <100ms from a warm pool.
+Execute scripts instantly on Magpie's microVM infrastructure. VMs boot in <100ms from a warm pool, giving you instant access to isolated compute environments.
+
+## What is Mags?
+
+Mags is a CLI and SDK for running scripts on ephemeral microVMs. Each execution gets its own isolated VM that:
+
+- Boots in ~300ms (from warm pool)
+- Supports optional S3-backed persistent workspaces (with `-p` flag)
+- Syncs /root directory automatically when persistence is enabled
+- Can expose public URLs for web services
+- Provides SSH access through secure proxy
+- Auto-sleeps when idle and wakes on request
 
 ## Installation
 
@@ -8,176 +19,411 @@ Execute scripts instantly on Magpie's microVM infrastructure. VMs boot in <100ms
 npm install -g @magpiecloud/mags
 ```
 
-## Quick Start
+## Authentication
 
-### 1. Login to Magpie
+### Interactive Login (Recommended)
 
 ```bash
 mags login
 ```
 
-This will open your browser to create an API token. Paste the token when prompted, and it will be saved for future use.
+This opens your browser to the Magpie dashboard where you can create an API token. Paste the token when prompted, and it will be saved to `~/.mags/config.json` for all future commands.
 
-### 2. Create a sandbox
+### Auth Commands
 
 ```bash
-mags new myproject              # Local disk only
-mags new myproject -p           # With S3 persistence
-mags ssh myproject
+mags login     # Authenticate with Magpie
+mags whoami    # Check current authentication status
+mags logout    # Remove saved credentials
 ```
 
-### 3. Or run a script
+### Environment Variable (Alternative)
 
 ```bash
-mags run 'echo Hello World'
+export MAGS_API_TOKEN="your-token-here"
 ```
 
-## Authentication
+## Quick Start
 
-### Interactive Login (Recommended)
+### Run a simple command
 
 ```bash
-mags login
+mags run 'echo Hello World'
 ```
 
-Opens the Magpie dashboard in your browser where you can create an API token. The token is saved to `~/.mags/config.json` and used automatically for all future commands.
-
-### Other Auth Commands
+### Create a sandbox
 
 ```bash
-mags whoami    # Check current authentication status
-mags logout    # Remove saved credentials
+# Create a new VM (local disk only)
+mags new myproject
+
+# Create with S3 data persistence
+mags new myproject -p
+
+# SSH into it
+mags ssh myproject
 ```
 
-### Environment Variable
+### Run with a persistent workspace
 
-You can also set the token via environment variable (overrides saved config):
+Workspaces persist data between runs using S3 sync:
 
 ```bash
-export MAGS_API_TOKEN="your-token-here"
+# First run - install dependencies
+mags run -w myproject 'apk add python3 py3-pip && pip install requests'
+
+# Second run - dependencies are still there
+mags run -w myproject 'python3 -c "import requests; print(requests.__version__)"'
 ```
 
-## CLI Commands
+### Deploy a web service with public URL
 
 ```bash
-# Create a sandbox (local disk)
-mags new myproject
+mags run -p --url 'python3 -m http.server 8080'
+# Returns: https://abc123.apps.magpiecloud.com
+```
 
-# Create with S3 persistence
-mags new myproject -p
+## CLI Reference
 
-# SSH into it
-mags ssh myproject
+### Commands
 
-# Run a simple command
-mags run 'echo Hello'
+| Command | Description |
+|---------|-------------|
+| `mags login` | Authenticate with Magpie (saves token locally) |
+| `mags logout` | Remove saved credentials |
+| `mags whoami` | Show current authentication status |
+| `mags new <workspace> [-p]` | Create a VM sandbox (add `-p` for S3 persistence) |
+| `mags run [options] <script>` | Execute a script on a microVM |
+| `mags ssh <workspace>` | Open interactive SSH session to a VM |
+| `mags status <job-id>` | Get job status |
+| `mags logs <job-id>` | Get job logs |
+| `mags list` | List recent jobs |
+| `mags url <job-id> [port]` | Enable URL access for a job |
+| `mags stop <job-id>` | Stop a running job |
+| `mags set <name\|id> [options]` | Update VM settings (`--no-sleep`, `--sleep`) |
 
-# With persistent workspace (S3 sync)
-mags run -w my-project 'apk add nodejs && node --version'
+### Run Options
 
-# Persistent VM with public URL
-mags run -w webapp -p --url 'python3 -m http.server 8080'
+| Flag | Description | Default |
+|------|-------------|---------|
+| `-w, --workspace <id>` | Use persistent workspace (S3 sync) | auto-generated |
+| `-p, --persistent` | Keep VM alive after script completes | false |
+| `--url` | Enable public URL access (requires -p) | false |
+| `--port <port>` | Port to expose for URL access | 8080 |
+| `--no-sleep` | Keep VM always running, never auto-sleep (requires -p) | false |
+| `--startup-command <cmd>` | Command to run when VM wakes from sleep | none |
+
+## SSH Access
 
-# With startup command (for auto-wake)
-mags run -w webapp -p --url --startup-command 'npm start' 'npm install && npm start'
+Connect to any running VM via SSH:
 
-# Custom port
-mags run -w webapp -p --url --port 3000 'npm start'
+```bash
+mags ssh myproject
+```
 
-# Enable URL for existing job
-mags url <job-id>
-mags url <job-id> 8080
+### SSH Features
 
-# Other commands
-mags status <job-id>
-mags logs <job-id>
-mags list
-mags stop <job-id>
+- **Secure proxy**: Connect via `api.magpiecloud.com:PORT` (agent IPs are hidden)
+- **PTY support**: Full interactive terminal with colors and editors
+- **Key-based auth**: Automatic SSH key management
+- **Hostname**: VMs use `mags-vm` as hostname
 
-# Install Claude Code skill
-mags setup-claude
+### SSH Session Example
+
+```bash
+$ mags ssh myproject
+Connecting to api.magpiecloud.com:40006...
+(Use Ctrl+D or 'exit' to disconnect)
+
+mags-vm:~# ls
+index.html  test.txt
+
+mags-vm:~# python3 --version
+Python 3.11.6
+
+mags-vm:~# exit
 ```
 
-## Claude Code Integration
+## Workspaces & Persistence
+
+### How It Works
+
+When you use a workspace, Mags:
 
-Install the Mags skill for Claude Code to run scripts directly from Claude:
+1. Mounts a JuiceFS filesystem backed by S3
+2. Creates an OverlayFS to capture all filesystem changes
+3. Syncs your `/root` directory to S3 every 30 seconds
+4. Restores your files when you reconnect
+
+### What Gets Persisted
+
+- `/root` - Your home directory (synced every 30 seconds)
+- `/jfs` - Direct JuiceFS mount (instant S3 sync)
+- Installed packages and dependencies
+- Configuration files and dotfiles
+
+### Sync Behavior
+
+| Event | Behavior |
+|-------|----------|
+| While running | Auto-sync every 30 seconds |
+| On stop | Full sync before VM terminates |
+| On wake | Previous state restored instantly |
+
+## Usage Examples
+
+### Basic Execution
 
 ```bash
-mags setup-claude
+# Simple command
+mags run 'echo Hello World'
+
+# Multi-line script
+mags run 'apk add curl && curl -s https://api.github.com/users/octocat | head -5'
+
+# With environment info
+mags run 'uname -a && cat /etc/os-release'
 ```
 
-This lets you use natural language commands like:
-- `/mags run echo Hello World`
-- `/mags create a python environment with numpy and pandas`
-- `/mags run a flask server and give me the public URL`
+### Persistent Workspaces
 
-See [mags.run/claude-skill.html](https://mags.run/claude-skill.html) for full documentation.
+```bash
+# Create a workspace with dependencies
+mags run -w ml-project 'apk add python3 py3-pip && pip install numpy pandas scikit-learn'
 
-## CLI Flags
+# Run scripts using the workspace
+mags run -w ml-project 'python3 train.py'
 
-| Flag | Description | Default |
-|------|-------------|---------|
-| `-w, --workspace` | Workspace ID for persistent storage | auto |
-| `-p, --persistent` | Keep VM alive for URL/SSH access | false |
-| `--url` | Enable public URL access (requires -p) | false |
-| `--port` | Port to expose for URL access | 8080 |
-| `--startup-command` | Command when VM wakes from sleep | none |
+# All files in /root are persisted
+mags run -w ml-project 'ls -la /root'
+```
 
-## SSH Access
+### Web Services
+
+```bash
+# Simple static server
+mags run -p --url 'python3 -m http.server 8080'
+
+# Node.js app
+mags run -w myapp -p --url --port 3000 'npm install && npm start'
+
+# Flask app
+mags run -w flask-app -p --url 'pip install flask && python app.py'
+
+# Custom startup command for auto-wake
+mags run -w api -p --url --startup-command 'python server.py' 'pip install -r requirements.txt && python server.py'
+```
 
-Connect to any running VM:
+### Always-On VMs
+
+By default, persistent VMs auto-sleep after 10 minutes of inactivity and wake on the next request. Use `--no-sleep` to keep a VM running 24/7:
 
 ```bash
-mags ssh myproject
+# Always-on API server (never auto-sleeps)
+mags run -w my-api -p --no-sleep --url --port 3000 'npm start'
+
+# Always-on background worker
+mags run -w worker -p --no-sleep 'python worker.py'
 ```
 
-Features:
-- Connects via secure proxy (`api.magpiecloud.com:PORT`)
-- Full PTY support for interactive terminals
-- Automatic SSH key management
-- Hostname: `mags-vm`
+If an always-on VM's host becomes unhealthy, the orchestrator automatically re-provisions it on a healthy agent within ~60 seconds.
 
-## Workspaces & Persistence
+### Interactive Development
 
-When using persistent mode (`-p`), your `/root` directory syncs to S3:
-- **Auto-sync**: Every 30 seconds while running
-- **On stop**: Full sync before VM terminates
-- **On wake**: Previous state restored
+```bash
+# Create a dev environment (local disk)
+mags new dev-env
+
+# SSH in and work
+mags ssh dev-env
 
-Without `-p`, data lives on local disk only and is cleaned up when the VM is destroyed.
+# Inside the VM:
+mags-vm:~# apk add git nodejs npm
+mags-vm:~# git clone https://github.com/user/repo
+mags-vm:~# cd repo && npm install
+mags-vm:~# npm run dev
+```
 
 ## Node.js SDK
 
+For programmatic access, use the SDK:
+
 ```javascript
 const Mags = require('@magpiecloud/mags');
 
 const mags = new Mags({
   apiToken: process.env.MAGS_API_TOKEN
 });
+```
+
+### SDK Methods
+
+#### Jobs
+
+| Method | Description |
+|--------|-------------|
+| `run(script, options)` | Submit a job. Options: `name`, `workspaceId`, `baseWorkspaceId`, `persistent`, `noSleep`, `ephemeral`, `startupCommand`, `environment`, `fileIds`, `diskGb` |
+| `runAndWait(script, options)` | Submit and block until done. Extra options: `timeout`, `pollInterval` |
+| `new(name, options)` | Create a VM sandbox and wait until running. Options: `persistent`, `baseWorkspaceId`, `diskGb`, `timeout` |
+| `status(requestId)` | Get job status |
+| `logs(requestId)` | Get job logs |
+| `list({page, pageSize})` | List recent jobs (paginated) |
+| `findJob(nameOrId)` | Find job by name, workspace ID, or request ID |
+| `updateJob(requestId, {startupCommand, noSleep})` | Update job settings |
+| `stop(nameOrId)` | Stop a job (accepts name, workspace ID, or request ID) |
+| `exec(nameOrId, command, {timeout})` | Execute a command on a running/sleeping VM via SSH |
+| `sync(requestId)` | Sync workspace to S3 without stopping |
+| `resize(workspace, diskGb)` | Resize workspace disk (stops VM, creates new one) |
+| `usage({windowDays})` | Get aggregated usage summary |
+
+#### URL & Access
+
+| Method | Description |
+|--------|-------------|
+| `enableAccess(requestId, port)` | Enable SSH (port 22) or HTTP access (default 8080) |
+| `url(nameOrId, port)` | Enable public URL and return the full URL |
+| `urlAliasCreate(subdomain, workspaceId, domain)` | Create a stable URL alias for a workspace |
+| `urlAliasList()` | List all URL aliases |
+| `urlAliasDelete(subdomain)` | Delete a URL alias |
+
+#### Workspaces
+
+| Method | Description |
+|--------|-------------|
+| `listWorkspaces()` | List all workspaces |
+| `deleteWorkspace(workspaceId)` | Delete a workspace and its S3 data |
+
+#### Files
+
+| Method | Description |
+|--------|-------------|
+| `uploadFiles(filePaths)` | Upload local files, returns array of file IDs |
+
+#### Cron Jobs
+
+| Method | Description |
+|--------|-------------|
+| `cronCreate({name, cronExpression, script, workspaceId, environment, persistent})` | Create a scheduled job |
+| `cronList()` | List all cron jobs |
+| `cronGet(id)` | Get a cron job |
+| `cronUpdate(id, updates)` | Update a cron job |
+| `cronDelete(id)` | Delete a cron job |
+
+### SDK Examples
 
+```javascript
 // Run and wait for completion
 const result = await mags.runAndWait('echo Hello World');
 console.log(result.logs);
 
-// Run with workspace
-const { requestId } = await mags.run('python script.py', {
+// Create a sandbox (local disk)
+await mags.new('myproject');
+
+// Create a sandbox with S3 persistence
+await mags.new('myproject', { persistent: true });
+
+// Execute command on existing VM
+const { output } = await mags.exec('myproject', 'ls -la /root');
+console.log(output);
+
+// Run with workspace and options
+const job = await mags.run('python script.py', {
   workspaceId: 'myproject',
-  persistent: true
+  persistent: true,
+  diskGb: 5,
+  startupCommand: 'python server.py'
+});
+
+// Always-on VM (never auto-sleeps)
+await mags.run('node server.js', {
+  workspaceId: 'my-api',
+  persistent: true,
+  noSleep: true
 });
 
-// Get status
-const status = await mags.status(requestId);
+// Enable public URL
+const { url } = await mags.url('my-api', 3000);
+console.log(url); // https://abc123.apps.magpiecloud.com
+
+// Create stable URL alias
+await mags.urlAliasCreate('my-app', 'my-api');
+
+// Resize workspace disk
+await mags.resize('myproject', 10); // 10GB
+
+// Stop a job by name
+await mags.stop('myproject');
+```
+
+## API Endpoints
+
+For direct API access:
 
-// Enable URL access
-await mags.enableUrl(requestId, 8080);
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/v1/mags-jobs` | POST | Submit a new job |
+| `/api/v1/mags-jobs` | GET | List jobs (paginated) |
+| `/api/v1/mags-jobs/{id}/status` | GET | Get job status |
+| `/api/v1/mags-jobs/{id}/logs` | GET | Get job logs |
+| `/api/v1/mags-jobs/{id}/access` | POST | Enable SSH/URL access |
+| `/api/v1/mags-jobs/{id}` | PATCH | Update job settings |
 
-// List jobs
-const jobs = await mags.list({ page: 1, pageSize: 10 });
+### Example API Request
 
-// Stop a job
-await mags.stop(requestId);
+```bash
+curl -X POST https://api.magpiecloud.com/api/v1/mags-jobs \
+  -H "Authorization: Bearer $MAGS_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "script": "echo Hello World",
+    "type": "inline",
+    "workspace_id": "myproject",
+    "persistent": true,
+    "no_sleep": true
+  }'
 ```
 
+## Job Status Values
+
+| Status | Description |
+|--------|-------------|
+| `pending` | Job queued, waiting for VM |
+| `running` | Script executing |
+| `sleeping` | Persistent VM idle (wakes on request) |
+| `completed` | Script finished successfully |
+| `error` | Script failed |
+
+## VM Environment
+
+Each VM runs Alpine Linux with hostname `mags-vm` and includes:
+
+- `/root` - Persistent home directory (synced to S3)
+- `/jfs` - Direct JuiceFS mount
+- Common tools: curl, wget, git, python3, nodejs
+- Package manager: `apk add <package>`
+
+### Installing Packages
+
+```bash
+# Python packages
+mags run 'apk add python3 py3-pip && pip install requests flask pandas'
+
+# Node.js packages
+mags run 'apk add nodejs npm && npm install -g express'
+
+# System packages
+mags run 'apk add ffmpeg imagemagick'
+```
+
+## Performance
+
+| Metric | Value |
+|--------|-------|
+| Warm start | ~300ms |
+| Cold start | ~4 seconds |
+| Script overhead | ~50ms |
+| Workspace sync | Every 30 seconds |
+
 ## Environment Variables
 
 | Variable | Description | Default |
@@ -185,12 +431,49 @@ await mags.stop(requestId);
 | `MAGS_API_TOKEN` | Your API token (required) | - |
 | `MAGS_API_URL` | API endpoint | https://api.magpiecloud.com |
 
-## Performance
+## Troubleshooting
+
+### "API token required"
+
+Set your API token:
+```bash
+mags login
+# or
+export MAGS_API_TOKEN="your-token-here"
+```
+
+### "Job timed out"
+
+Increase the timeout or check if your script is hanging:
+```bash
+mags run --timeout 600 'long-running-script.sh'
+```
+
+### "Workspace not found"
+
+Workspace IDs are case-sensitive. Check with:
+```bash
+mags list
+```
+
+### URL not accessible
+
+Make sure you're using both `-p` (persistent) and `--url` flags, and your app is listening on the correct port:
+```bash
+mags run -p --url --port 3000 'node server.js'
+```
+
+### SSH connection issues
+
+SSH connects through our proxy. If you have issues:
+1. Ensure the job is running: `mags status <id>`
+2. Try reconnecting: `mags ssh <workspace>`
+
+## Support
 
-- **Warm start**: <100ms (VM from pool)
-- **Cold start**: ~4 seconds (new VM boot)
-- **Script overhead**: ~50ms
-- **Workspace sync**: Every 30 seconds
+- Documentation: https://mags.run
+- API Reference: https://mags.run/api.html
+- Dashboard: https://mags.run/tokens.html
 
 ## License
 

package/bin/mags.js

@@ -230,6 +230,7 @@ ${colors.bold}Run Options:${colors.reset}
   --url                           Enable public URL access (requires -p)
   --port <port>                   Port to expose for URL (default: 8080)
   --startup-command <cmd>         Command to run when VM wakes from sleep
+  -t, --type <type>               VM rootfs type: standard, claude, pi
 
 ${colors.bold}Cron Commands:${colors.reset}
   cron add [options] <script>     Create a scheduled cron job
@@ -409,6 +410,7 @@ async function newVM(args) {
   let baseWorkspace = null;
   let diskGB = 0;
   let persistent = false;
+  let rootfsType = '';
 
   for (let i = 0; i < args.length; i++) {
     if (args[i] === '-p' || args[i] === '--persistent') {
@@ -417,6 +419,8 @@ async function newVM(args) {
       baseWorkspace = args[++i];
     } else if (args[i] === '--disk' && args[i + 1]) {
       diskGB = parseInt(args[++i]) || 0;
+    } else if ((args[i] === '--type' || args[i] === '-t') && args[i + 1]) {
+      rootfsType = args[++i];
     } else if (!name) {
       name = args[i];
     }
@@ -424,8 +428,9 @@ async function newVM(args) {
 
   if (!name) {
     log('red', 'Error: Name required');
-    console.log(`\nUsage: mags new <name> [-p] [--base <workspace>] [--disk <GB>]`);
-    console.log(`  -p, --persistent  Enable S3 data persistence\n`);
+    console.log(`\nUsage: mags new <name> [-p] [--base <workspace>] [--disk <GB>] [--type <type>]`);
+    console.log(`  -p, --persistent  Enable S3 data persistence`);
+    console.log(`  -t, --type        VM type: standard (default), claude, pi\n`);
     process.exit(1);
   }
 
@@ -457,6 +462,7 @@ async function newVM(args) {
   };
   if (baseWorkspace) payload.base_workspace_id = baseWorkspace;
   if (diskGB) payload.disk_gb = diskGB;
+  if (rootfsType) payload.rootfs_type = rootfsType;
 
   const response = await request('POST', '/api/v1/mags-jobs', payload);
 
@@ -506,6 +512,7 @@ async function runJob(args) {
   let startupCommand = '';
   let diskGB = 0;
   let fileArgs = [];
+  let rootfsType = '';
 
   // Parse flags
   for (let i = 0; i < args.length; i++) {
@@ -546,6 +553,10 @@ async function runJob(args) {
       case '--startup-command':
         startupCommand = args[++i];
         break;
+      case '--type':
+      case '-t':
+        rootfsType = args[++i];
+        break;
       default:
         script = args.slice(i).join(' ');
         i = args.length;
@@ -608,6 +619,7 @@ async function runJob(args) {
   if (startupCommand) payload.startup_command = startupCommand;
   if (fileIds.length > 0) payload.file_ids = fileIds;
   if (diskGB) payload.disk_gb = diskGB;
+  if (rootfsType) payload.rootfs_type = rootfsType;
 
   const response = await request('POST', '/api/v1/mags-jobs', payload);
 
@@ -1508,6 +1520,11 @@ async function proxyTunnel(jobID) {
 
   ws.on('open', () => {
     // Don't flush yet — wait for first message (SSH key)
+    // Send pings every 30s to keep the connection alive through proxies/CDNs
+    const pingInterval = setInterval(() => {
+      if (ws.readyState === WebSocket.OPEN) ws.ping();
+    }, 30000);
+    ws.on('close', () => clearInterval(pingInterval));
   });
 
   ws.on('message', (data, isBinary) => {
@@ -1672,6 +1689,8 @@ async function sshToJob(nameOrId) {
     '-o', 'StrictHostKeyChecking=no',
     '-o', 'UserKnownHostsFile=/dev/null',
     '-o', 'LogLevel=ERROR',
+    '-o', 'ServerAliveInterval=15',
+    '-o', 'ServerAliveCountMax=4',
     '-o', `ProxyCommand=${magsPath} proxy ${jobID}`,
     '-i', keyFile,
     'root@mags-vm',
@@ -1739,6 +1758,8 @@ async function execOnJob(nameOrId, command) {
     '-o', 'StrictHostKeyChecking=no',
     '-o', 'UserKnownHostsFile=/dev/null',
     '-o', 'LogLevel=ERROR',
+    '-o', 'ServerAliveInterval=15',
+    '-o', 'ServerAliveCountMax=4',
     '-o', `ProxyCommand=${magsPath} proxy ${jobID}`,
     '-i', keyFile,
   ];