@cloud-cli/on 0.1.7 → 1.2.2

package/README.md

@@ -0,0 +1,241 @@
+# 🏃 `@cloud-cli/on`
+
+General-purpose workflows
+
+A self-hosted, lightweight, high-performance CI/CD runner engine built for Node.js.
+
+Designed with a strict **security-first boundary**, native **JavaScript AST evaluation**, zero-DSL template literals, and a **built-in terminal log web dashboard**.
+
+---
+
+## 🌟 Key Highlights
+
+- **Strict Code/Data Separation:** `run:` steps are executed verbatim as raw process scripts. Expressions and dynamic data bindings are isolated strictly to `env:`, eliminating shell-injection vectors entirely.
+- **Standard ES Template Syntax (`${...}`):** No custom DSL wrappers like `${{ }}` or `{{ }}`. If a field contains `${...}`, it evaluates standard JavaScript template string logic via AST.
+- **Deterministic Field Evaluation:** No silent fallbacks or ambiguous type conversions. Plain strings remain literal strings; conditions in `if:` fields run as strict JS boolean expressions.
+- **Built-in Dark Mode Web UI (`/runs`):** Monitor job statuses live, inspect workspace inputs, and view ANSI-colored terminal log streams rendered in real-time.
+- **System & Container Execution Drivers:** Run steps directly as detached host process groups or inside isolated Docker/Systemd transient units.
+- **Automatic Secret Redaction:** Secrets loaded from `.env` are automatically masked (`***`) across all terminal log outputs and report snapshots.
+
+---
+
+## 📦 Project Structure
+
+```text
+my-project/
+├── .on/                     # Workflow definitions directory
+│   ├── release.yml
+│   └── test.yml
+├── .env                     # Local secrets (git-ignored)
+├── runner.config.mjs        # (Optional) Engine configuration
+└── package.json
+
+```
+
+---
+
+## 🚀 Quick Start
+
+### 1. Install & Run
+
+Run the engine directly via `npx` or `pnpm dlx`:
+
+```bash
+# Start full engine (Ingress HTTP Gateway + 5 Worker Loops)
+npx @cloud-cli/on start
+
+```
+
+### 2. Configure Secrets (`.env`)
+
+Secrets are automatically loaded from `.env` at the root of your project. Prefix secrets with `SECRET_`:
+
+```env
+SECRET_NPM_TOKEN="npm_1234567890abcdef"
+SECRET_GITHUB_TOKEN="ghp_1234567890abcdef"
+SECRET_GITHUB_WEBHOOK_SECRET="my-webhook-secret"
+
+```
+
+### 3. Define a Workflow (`.on/release.yml`)
+
+```yaml
+name: Build and Publish Release
+
+on:
+  github:
+    if: inputs.event === 'push' && inputs.branch === 'main'
+
+concurrency:
+  group: release-${inputs.repo}
+  cancel-in-progress: true
+
+steps:
+  - id: checkout
+    name: Checkout Code Repository
+    env:
+      CLONE_URL: ${inputs.clone_url}
+      COMMIT_SHA: ${inputs.commit_sha}
+    run: |
+      git clone --depth 1 "$CLONE_URL" .
+      git checkout "$COMMIT_SHA"
+
+  - id: install-and-build
+    name: Install Dependencies & Build
+    run: |
+      pnpm install
+      pnpm run build
+
+  - id: publish
+    name: Publish to NPM
+    env:
+      NPM_TOKEN: ${secrets.NPM_TOKEN}
+    run: |
+      echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/.npmrc
+      npx --yes semantic-release@24 -b main --no-ci
+```
+
+---
+
+## 💻 CLI Usage & Commands
+
+```bash
+npx @cloud-cli/on [command] [options]
+
+```
+
+### Commands
+
+| Command                 | Description                                                               |
+| ----------------------- | ------------------------------------------------------------------------- |
+| **`start`** _(default)_ | Runs both Webhook Ingress Gateway and Worker execution loops together.    |
+| **`start-server`**      | Runs Webhook Ingress Gateway only (API / Gateway mode).                   |
+| **`start-workers`**     | Runs Worker Polling loops only (Scalable Worker mode).                    |
+| **`validate`**          | Parses and validates all YAML workflows in `.on/` without executing jobs. |
+
+### CLI Options
+
+| Flag | Option        | Default                    | Description                               |
+| ---- | ------------- | -------------------------- | ----------------------------------------- |
+| `-c` | `--config`    | `./runner.config.mjs`      | Path to JavaScript configuration file.    |
+| `-d` | `--database`  | `process.env.DATABASE_URL` | SQLite database file path or HTTP URL.    |
+| `-w` | `--workflows` | `.on/`                     | Directory where workflow YAML files live. |
+| `-p` | `--port`      | `3000`                     | Port for the Ingress HTTP server.         |
+| `-k` | `--workers`   | `5`                        | Number of worker loop threads to spawn.   |
+| `-h` | `--help`      | —                          | Prints CLI help message and exits.        |
+
+---
+
+## ⚙️ Configuration Reference
+
+You can customize engine behavior using `runner.config.mjs` in your project root:
+
+```javascript
+// runner.config.mjs
+import { HtmlReporter, SlackReporter, JsonFileReporter } from '@cloud-cli/on/reporters';
+
+export default {
+  port: 3000,
+  workersCount: 5,
+  workflowsDir: '.on/',
+  storagePath: '/tmp/workspaces',
+  sqliteUrl: 'sqlite.db',
+
+  // Global environment variables passed to all steps
+  env: {
+    NODE_ENV: 'production',
+  },
+
+  // Custom execution reporters
+  reporters: [
+    new JsonFileReporter({ outputDir: './reports/json' }),
+    new HtmlReporter({ outputDir: './reports/html' }),
+    new SlackReporter({
+      webhookUrl: process.env.SLACK_WEBHOOK_URL,
+      channel: '#ci-deployments',
+    }),
+  ],
+};
+```
+
+---
+
+## 📐 Deterministic Evaluation Rules
+
+To prevent syntax ambiguity and injection risks, fields in workflow definitions operate under **three strict modes**:
+
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│ 1. RAW PASSTHROUGH MODE (`run:`)                                         │
+│ • Executed verbatim as a shell process command.                          │
+│ • No string replacements or engine parsing performed.                    │
+│ • Access environment variables strictly via shell syntax: $MY_VAR.       │
+└──────────────────────────────────────────────────────────────────────────┘
+
+┌──────────────────────────────────────────────────────────────────────────┐
+│ 2. EXPRESSION MODE (`if:`, `eval:`)                                      │
+│ • Evaluated strictly as pure JavaScript expressions via Acorn AST.       │
+│ • Must be valid JS syntax (e.g. `inputs.branch === 'main'`).             │
+│ • Automatically coerced to boolean in `if:` conditions.                  │
+└──────────────────────────────────────────────────────────────────────────┘
+
+┌──────────────────────────────────────────────────────────────────────────┐
+│ 3. DETERMINISTIC VALUE MODE (`env:`, `name:`, `image:`, `group:`)        │
+│ • Plain strings WITHOUT `${}` remain 100% raw literal strings.            │
+│ • Strings WITH `${...}` evaluate as standard ES Template Literals.       │
+│ • Example: `node:${inputs.node_version}-alpine`                          │
+└──────────────────────────────────────────────────────────────────────────┘
+
+```
+
+### Context Scope Available in Expressions
+
+Within `${...}`, `if:`, and `eval:` contexts, the following object scopes are exposed:
+
+- **`inputs`**: Payload key-values received from incoming webhooks.
+- **`env`**: Merged environment variables from global config and workflow definitions.
+- **`secrets`**: Unmasked secret values loaded from `.env` (`SECRET_` prefix stripped).
+- **`steps`**: Execution statuses and outputs from previous steps in the workflow (`steps.<id>.status`, `steps.<id>.outputs`).
+- **`BUILTIN_HELPERS`**: JS utilities including `String`, `Number`, `Boolean`, and `JSON.parse` / `JSON.stringify`.
+
+---
+
+## 🌐 Webhook Ingress Gateway & Dashboard
+
+The Ingress Gateway listens for incoming HTTP requests and serves the live web UI.
+
+### Endpoint Matrix
+
+| Method     | Endpoint           | Description                                                                            |
+| ---------- | ------------------ | -------------------------------------------------------------------------------------- |
+| **`POST`** | `/webhooks/github` | Webhook endpoint for GitHub events. Evaluates `on.github.if` triggers.                 |
+| **`GET`**  | `/runs`            | **Dashboard:** Live dark-mode monitoring page listing recent jobs and worker health.   |
+| **`GET`**  | `/runs/:jobId`     | **Job Report:** Interactive HTML trace view with step timings and terminal log output. |
+
+### Dashboard Features
+
+- **Real-time Auto-Refresh:** `/runs` automatically refreshes job queue statuses (`PENDING`, `RUNNING`, `SUCCESS`, `FAILED`, `CANCELLED`).
+- **ANSI Terminal Rendering:** Uses `ansi_up` to render bash colors, bold highlights, and console outputs accurately in step log boxes.
+- **Payload Inspection:** View JSON inputs received from webhooks for easy debugging.
+
+---
+
+## 🔒 Security & Hardening
+
+1. **Environment Variable Boundary:**
+   By forcing shell steps to consume data via process environment variables (`$CLONE_URL`), malicious webhook payloads containing shell delimiters (e.g. `; rm -rf /`) cannot mutate shell script execution trees.
+2. **Prototype Pollution Protection:**
+   AST evaluation explicitly blocks access to dangerous JS properties (`constructor`, `__proto__`, `prototype`).
+3. **Payload Size Guard:**
+   The Ingress server enforces a strict 5MB payload limit to prevent Out-Of-Memory (OOM) denial-of-service attacks.
+4. **Signal Traps & Resource Cleanup:**
+   Graceful process traps (`SIGINT`, `SIGTERM`) ensure active job handles are safely terminated, file descriptors are closed, and temp `.env`/`.out` files are removed via `try ... finally` blocks.
+
+## Development
+
+```bash
+pnpm i
+pnpm run lint
+pnpm run test
+pnpm run build
+```

package/dist/config.js

@@ -0,0 +1,15 @@
+/**
+ * Merges user config overrides with baseline defaults
+ */
+export function resolveConfig(userConfig = {}) {
+    return {
+        port: userConfig.port ?? 3000,
+        adminToken: userConfig.adminToken ?? process.env.RUNNER_ADMIN_SECRET ?? '',
+        sqliteUrl: userConfig.sqliteUrl ?? process.env.DATABASE_URL ?? 'sqlite.db',
+        workflowsDir: userConfig.workflowsDir ?? '.on/',
+        workersCount: userConfig.workersCount ?? 5,
+        storagePath: userConfig.storagePath ?? process.env.RUNNER_TMP ?? '/tmp/workspaces',
+        env: userConfig.env ?? {},
+        reporters: userConfig.reporters ?? [],
+    };
+}

package/dist/db-client.js

@@ -0,0 +1,38 @@
+const baseURL = process.env.DATABASE_URL;
+let pragmas = [];
+async function query(method, statement, data, pragma = pragmas) {
+    let req;
+    let error;
+    let retries = 1;
+    let max = 3;
+    while (retries < max) {
+        try {
+            req = await fetch(new URL('/query', baseURL), {
+                method: 'POST',
+                body: JSON.stringify({
+                    s: statement,
+                    d: data,
+                    m: method,
+                    p: pragma,
+                }),
+            });
+            if (req.ok) {
+                return await req.json();
+            }
+            await new Promise((r) => setTimeout(r, retries++ * 1000));
+        }
+        catch (e) {
+            error = e;
+        }
+    }
+    throw new Error(error || (await req.text()));
+}
+export const get = query.bind(null, 'get');
+export const run = query.bind(null, 'run');
+export const all = query.bind(null, 'all');
+export function pragma(p) {
+    if (Array.isArray(p) && p.every((s) => typeof s === 'string')) {
+        pragmas = p;
+    }
+}
+export default { query, get, run, all, pragma };

package/dist/drivers/index.js

@@ -0,0 +1,11 @@
+import { SystemdDriver } from './systemd.driver.js';
+import { StandardProcessDriver } from './standard-process.driver.js';
+export async function resolveDriver() {
+    const systemd = new SystemdDriver();
+    if (await systemd.isSupported()) {
+        console.log('⚡ Selected Execution Driver: Systemd (cgroups enabled)');
+        return systemd;
+    }
+    console.log('📦 Selected Execution Driver: Standard Process (Fallback)');
+    return new StandardProcessDriver();
+}

package/dist/drivers/standard-process.driver.js

@@ -0,0 +1,156 @@
+import { spawn } from 'node:child_process';
+import fs from 'node:fs';
+import path from 'node:path';
+export class StandardProcessDriver {
+    name = 'standard-process';
+    async isSupported() {
+        return true; // Supported on all OS platforms
+    }
+    async execute(ctx) {
+        const startTime = Date.now();
+        let logFd = null;
+        let logFilePath = '';
+        // 1. Guard Log Directory & File Handle Creation
+        try {
+            const logDir = path.join(ctx.workspacePath, '.logs');
+            fs.mkdirSync(logDir, { recursive: true });
+            logFilePath = path.join(logDir, `step-${ctx.stepId}.log`);
+            logFd = fs.openSync(logFilePath, 'a');
+        }
+        catch (err) {
+            return {
+                done: Promise.resolve({
+                    exitCode: 1,
+                    durationMs: 0,
+                    error: new Error(`Failed to initialize step log file: ${err.message}`),
+                }),
+                cancel: async () => { },
+                logFilePath: '',
+            };
+        }
+        // 2. Format Execution Command
+        let cmd;
+        let args;
+        if (ctx.image) {
+            cmd = 'docker';
+            args = [
+                'run',
+                '--rm',
+                '--init',
+                '-v',
+                `${ctx.workspacePath}:/workspace`,
+                '-w',
+                '/workspace',
+                '--entrypoint',
+                'sh',
+                ctx.image,
+                '-c',
+                ctx.command,
+            ];
+        }
+        else {
+            cmd = 'sh';
+            args = ['-c', ctx.command];
+        }
+        // 3. Spawn Detached Child Process
+        let child;
+        try {
+            child = spawn(cmd, args, {
+                cwd: ctx.workspacePath,
+                env: { ...process.env, ...ctx.env },
+                detached: true, // Creates separate Process Group ID
+                stdio: ['ignore', logFd, logFd],
+            });
+        }
+        catch (spawnErr) {
+            try {
+                if (logFd !== null)
+                    fs.closeSync(logFd);
+            }
+            catch { }
+            return {
+                done: Promise.resolve({
+                    exitCode: 1,
+                    durationMs: Date.now() - startTime,
+                    error: new Error(`Failed to spawn process: ${spawnErr.message}`),
+                }),
+                cancel: async () => { },
+                logFilePath,
+            };
+        }
+        let isCancelled = false;
+        let timeoutTimer = null;
+        // 4. Safe Promise Resolution & Lifecycle Tracking
+        const done = new Promise((resolve) => {
+            let isResolved = false;
+            const safeResolve = (result) => {
+                if (isResolved)
+                    return; // Prevent double-resolution
+                isResolved = true;
+                if (timeoutTimer)
+                    clearTimeout(timeoutTimer);
+                // Always close log File Descriptor safely
+                try {
+                    if (logFd !== null)
+                        fs.closeSync(logFd);
+                }
+                catch { }
+                resolve(result);
+            };
+            // Optional step timeout
+            if (ctx.timeoutMs) {
+                timeoutTimer = setTimeout(() => {
+                    isCancelled = true;
+                    this.killProcessGroup(child);
+                }, ctx.timeoutMs);
+                // CRUCIAL: Unref timer so it doesn't hold event loop open
+                timeoutTimer.unref();
+            }
+            child.on('close', (code) => {
+                safeResolve({
+                    exitCode: code ?? (isCancelled ? 130 : 1),
+                    durationMs: Date.now() - startTime,
+                    error: isCancelled ? new Error('Step timed out or was cancelled by user') : undefined,
+                });
+            });
+            child.on('error', (err) => {
+                safeResolve({
+                    exitCode: 1,
+                    durationMs: Date.now() - startTime,
+                    error: err,
+                });
+            });
+        });
+        // 5. Cancellation Hook
+        const cancel = async () => {
+            isCancelled = true;
+            this.killProcessGroup(child);
+        };
+        return { done, cancel, logFilePath };
+    }
+    /**
+     * Kills the entire process group tree (-PID) with unref escalation
+     */
+    killProcessGroup(child) {
+        if (child.pid && !child.killed) {
+            try {
+                // Send SIGTERM to entire process group (-PID)
+                process.kill(-child.pid, 'SIGTERM');
+                // Escalate to SIGKILL after 5 seconds if process tree is still alive
+                const killTimer = setTimeout(() => {
+                    try {
+                        if (child.pid && !child.killed) {
+                            process.kill(-child.pid, 'SIGKILL');
+                        }
+                    }
+                    catch { }
+                }, 5000);
+                // CRUCIAL: Unref escalation timer so Node process can exit cleanly
+                killTimer.unref();
+            }
+            catch {
+                // Process group may already be dead
+            }
+        }
+    }
+}

package/dist/drivers/systemd.driver.js

@@ -0,0 +1,151 @@
+import { spawn, exec } from 'node:child_process';
+import fs from 'node:fs';
+import path from 'node:path';
+import { promisify } from 'node:util';
+const execAsync = promisify(exec);
+export class SystemdDriver {
+    name = 'systemd';
+    /**
+     * Check if systemd bus is available on Linux host
+     */
+    async isSupported() {
+        try {
+            return fs.existsSync('/run/systemd/system');
+        }
+        catch {
+            return false;
+        }
+    }
+    async execute(ctx) {
+        const startTime = Date.now();
+        let logFd = null;
+        let logFilePath = '';
+        // 1. Guard Log Directory & File Handle Creation
+        try {
+            const logDir = path.join(ctx.workspacePath, '.logs');
+            fs.mkdirSync(logDir, { recursive: true });
+            logFilePath = path.join(logDir, `step-${ctx.stepId}.log`);
+            logFd = fs.openSync(logFilePath, 'a');
+        }
+        catch (err) {
+            return {
+                done: Promise.resolve({
+                    exitCode: 1,
+                    durationMs: 0,
+                    error: new Error(`Failed to initialize step log file: ${err.message}`),
+                }),
+                cancel: async () => { },
+                logFilePath: '',
+            };
+        }
+        // 2. Format Sanitized Systemd Unit Name
+        const unitName = `workflow-${ctx.jobId}-${ctx.stepId}`.replace(/[^a-zA-Z0-9_-]/g, '_');
+        // 3. Build systemd-run Flags
+        const systemdFlags = [
+            `--unit=${unitName}`,
+            '--wait', // Block until unit completes
+            '--pipe', // Stream stdio directly to file handle
+            `--working-directory=${ctx.workspacePath}`,
+        ];
+        if (ctx.env) {
+            for (const [key, val] of Object.entries(ctx.env)) {
+                systemdFlags.push(`--setenv=${key}=${val}`);
+            }
+        }
+        if (ctx.timeoutMs) {
+            const timeoutSec = Math.ceil(ctx.timeoutMs / 1000);
+            systemdFlags.push(`--property=RuntimeMaxSec=${timeoutSec}`);
+        }
+        // 4. Construct Command
+        let commandArgs;
+        if (ctx.image) {
+            commandArgs = [
+                'docker',
+                'run',
+                '--rm',
+                '--init',
+                `--name=${unitName}`, // Predictable container name for stopping
+                '-v',
+                `${ctx.workspacePath}:/workspace`,
+                '-w',
+                '/workspace',
+                ctx.image,
+                'sh',
+                '-c',
+                ctx.command,
+            ];
+        }
+        else {
+            commandArgs = ['sh', '-c', ctx.command];
+        }
+        // 5. Spawn systemd-run
+        let child;
+        try {
+            child = spawn('systemd-run', [...systemdFlags, '--', ...commandArgs], {
+                stdio: ['ignore', logFd, logFd],
+            });
+        }
+        catch (spawnErr) {
+            try {
+                if (logFd !== null)
+                    fs.closeSync(logFd);
+            }
+            catch { }
+            return {
+                done: Promise.resolve({
+                    exitCode: 1,
+                    durationMs: Date.now() - startTime,
+                    error: new Error(`Failed to spawn systemd-run: ${spawnErr.message}`),
+                }),
+                cancel: async () => { },
+                logFilePath,
+            };
+        }
+        let isCancelled = false;
+        // 6. Safe Promise Resolution & File Handle Cleanup
+        const done = new Promise((resolve) => {
+            let isResolved = false;
+            const safeResolve = (result) => {
+                if (isResolved)
+                    return; // Prevent double-resolution
+                isResolved = true;
+                try {
+                    if (logFd !== null)
+                        fs.closeSync(logFd);
+                }
+                catch { }
+                resolve(result);
+            };
+            child.on('close', (code) => {
+                safeResolve({
+                    exitCode: code ?? (isCancelled ? 130 : 1),
+                    durationMs: Date.now() - startTime,
+                    error: isCancelled ? new Error('Step cancelled by user or systemd timeout') : undefined,
+                });
+            });
+            child.on('error', (err) => {
+                safeResolve({
+                    exitCode: 1,
+                    durationMs: Date.now() - startTime,
+                    error: err,
+                });
+            });
+        });
+        // 7. Systemd / Docker Graceful Cancellation
+        const cancel = async () => {
+            isCancelled = true;
+            try {
+                if (ctx.image) {
+                    // Stop docker container gracefully if running
+                    await execAsync(`docker stop -t 2 ${unitName}`).catch(() => { });
+                }
+                // Stop systemd transient unit (sends SIGTERM -> SIGKILL to Cgroup tree)
+                await execAsync(`systemctl stop ${unitName}.service`).catch(() => { });
+            }
+            catch {
+                // Unit or container may already be stopped
+            }
+        };
+        return { done, cancel, logFilePath };
+    }
+}