@cloud-cli/on 0.1.9 → 1.2.2

package/README.md

@@ -1,180 +1,241 @@
-# Workflow Design
+# 🏃 `@cloud-cli/on`
 
-This is a task runner using webhooks to process incoming events.
+General-purpose workflows
 
-Every event is sent to a daemon as an HTTP request, with a JSON body, and can trigger one or more workflows.
-Every workflow is a set of steps, which can run on containers or in a shell on the host.
+A self-hosted, lightweight, high-performance CI/CD runner engine built for Node.js.
 
-## Running on Docker
+Designed with a strict **security-first boundary**, native **JavaScript AST evaluation**, zero-DSL template literals, and a **built-in terminal log web dashboard**.
 
-- Steps run inside a docker container.
-- All steps run in the same workspace folder.
-- The current folder is mounted as a volume at /workspace by default. This can be changed by specifying a volume with `.` as the host path.
+---
 
-## Running on the host machine
+## 🌟 Key Highlights
 
-- Steps are sent as stdin to a shell subprocess
+- **Strict Code/Data Separation:** `run:` steps are executed verbatim as raw process scripts. Expressions and dynamic data bindings are isolated strictly to `env:`, eliminating shell-injection vectors entirely.
+- **Standard ES Template Syntax (`${...}`):** No custom DSL wrappers like `${{ }}` or `{{ }}`. If a field contains `${...}`, it evaluates standard JavaScript template string logic via AST.
+- **Deterministic Field Evaluation:** No silent fallbacks or ambiguous type conversions. Plain strings remain literal strings; conditions in `if:` fields run as strict JS boolean expressions.
+- **Built-in Dark Mode Web UI (`/runs`):** Monitor job statuses live, inspect workspace inputs, and view ANSI-colored terminal log streams rendered in real-time.
+- **System & Container Execution Drivers:** Run steps directly as detached host process groups or inside isolated Docker/Systemd transient units.
+- **Automatic Secret Redaction:** Secrets loaded from `.env` are automatically masked (`***`) across all terminal log outputs and report snapshots.
 
-## General configuration syntax
+---
 
-Running with Docker:
+## 📦 Project Structure
+
+```text
+my-project/
+├── .on/                     # Workflow definitions directory
+│   ├── release.yml
+│   └── test.yml
+├── .env                     # Local secrets (git-ignored)
+├── runner.config.mjs        # (Optional) Engine configuration
+└── package.json
 
-```sh
-curl -X POST http://localhost:11235/ -d '{ "event-name": {...} }'
 ```
 
-```yaml
-description: Run tests and build
+---
 
-vars: &vars
-  image: node:latest
+## 🚀 Quick Start
 
-on:
-  event-name:
-    runner: docker
-    if:
-      - ${inputs.action} == 'published'
-    secrets:
-      - /path/to/secrets
-      - /path/to/.env
-    mappings:
-      <field>: <path.to.value.in.inputs>
-    env:
-      A_SECRET: "${secrets.A_SECRET}"
-      A_VALUE: "${inputs.some.value}"
-    defaults:
-      <<: *vars
-      volumes:
-        .: /home
-        /dev/shm: /dev/shm
-      args:
-        net: host
-        dns: 1.2.3.4
-    steps:
-      - pnpm i
-      - pnpm run build
-      - pnpm run test
-    triggers:
-      - path/to/output.json
-```
+### 1. Install & Run
 
-Running with a shell on the same machine as the server:
+Run the engine directly via `npx` or `pnpm dlx`:
 
-```sh
-curl -X POST http://localhost:11235/ -d '{ "package": {...} }'
-```
+```bash
+# Start full engine (Ingress HTTP Gateway + 5 Worker Loops)
+npx @cloud-cli/on start
 
-```yaml
-description: Auto-release library
-on:
-  event-name:
-    runner: shell
-    secrets:
-      - /path/to/secrets
-      - /path/to/.env
-    mappings:
-      <field>: <path.to.value.in.json.payload>
-    env:
-      A_SECRET: "${secrets.A_SECRET}"
-      A_VALUE: "${inputs.some.value}"
-    steps:
-      - pnpm i
-      - pnpm run build
-      - pnpm run release
-    triggers:
-      - path/to/output.json
 ```
 
-### Event Payload
+### 2. Configure Secrets (`.env`)
 
-The webhooks can have any shape. To match events to workflows, we look for the top-level keys in the request body, and matching workflows that expect them to be present.
+Secrets are automatically loaded from `.env` at the root of your project. Prefix secrets with `SECRET_`:
 
-For example, given the request:
+```env
+SECRET_NPM_TOKEN="npm_1234567890abcdef"
+SECRET_GITHUB_TOKEN="ghp_1234567890abcdef"
+SECRET_GITHUB_WEBHOOK_SECRET="my-webhook-secret"
 
-```sh
-curl -X POST http://localhost:11235/ -d '{ "published": { "value" : 123 } }'
 ```
 
-Then the workflow should map the `published` key:
+### 3. Define a Workflow (`.on/release.yml`)
 
 ```yaml
+name: Build and Publish Release
+
 on:
-  published:
-    steps:
-      - echo ${inputs.value}
+  github:
+    if: inputs.event === 'push' && inputs.branch === 'main'
+
+concurrency:
+  group: release-${inputs.repo}
+  cancel-in-progress: true
+
+steps:
+  - id: checkout
+    name: Checkout Code Repository
+    env:
+      CLONE_URL: ${inputs.clone_url}
+      COMMIT_SHA: ${inputs.commit_sha}
+    run: |
+      git clone --depth 1 "$CLONE_URL" .
+      git checkout "$COMMIT_SHA"
+
+  - id: install-and-build
+    name: Install Dependencies & Build
+    run: |
+      pnpm install
+      pnpm run build
+
+  - id: publish
+    name: Publish to NPM
+    env:
+      NPM_TOKEN: ${secrets.NPM_TOKEN}
+    run: |
+      echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/.npmrc
+      npx --yes semantic-release@24 -b main --no-ci
 ```
 
-Here, the expression `inputs` in the workflow context is defined as the value set in the `published` key from the parsed JSON.
-The expression `${inputs.value}` contains `123`.
+---
 
-To differentiate between events coming from the same place, multiple webhooks can be created.
-The incoming webhook URL can have a path, and that is considered as the event source.
+## 💻 CLI Usage & Commands
 
-Consider this request:
+```bash
+npx @cloud-cli/on [command] [options]
 
-```sh
-curl -X POST http://localhost:11235/source -d '{ "event": { "value" : 123 } }'
 ```
 
-The key `source` is added to the event payload. The workflow should now be defined as:
-
-```yaml
-on:
-  source:
-    published:
-      steps:
-        - echo ${inputs.value}
+### Commands
+
+| Command                 | Description                                                               |
+| ----------------------- | ------------------------------------------------------------------------- |
+| **`start`** _(default)_ | Runs both Webhook Ingress Gateway and Worker execution loops together.    |
+| **`start-server`**      | Runs Webhook Ingress Gateway only (API / Gateway mode).                   |
+| **`start-workers`**     | Runs Worker Polling loops only (Scalable Worker mode).                    |
+| **`validate`**          | Parses and validates all YAML workflows in `.on/` without executing jobs. |
+
+### CLI Options
+
+| Flag | Option        | Default                    | Description                               |
+| ---- | ------------- | -------------------------- | ----------------------------------------- |
+| `-c` | `--config`    | `./runner.config.mjs`      | Path to JavaScript configuration file.    |
+| `-d` | `--database`  | `process.env.DATABASE_URL` | SQLite database file path or HTTP URL.    |
+| `-w` | `--workflows` | `.on/`                     | Directory where workflow YAML files live. |
+| `-p` | `--port`      | `3000`                     | Port for the Ingress HTTP server.         |
+| `-k` | `--workers`   | `5`                        | Number of worker loop threads to spawn.   |
+| `-h` | `--help`      | —                          | Prints CLI help message and exits.        |
+
+---
+
+## ⚙️ Configuration Reference
+
+You can customize engine behavior using `runner.config.mjs` in your project root:
+
+```javascript
+// runner.config.mjs
+import { HtmlReporter, SlackReporter, JsonFileReporter } from '@cloud-cli/on/reporters';
+
+export default {
+  port: 3000,
+  workersCount: 5,
+  workflowsDir: '.on/',
+  storagePath: '/tmp/workspaces',
+  sqliteUrl: 'sqlite.db',
+
+  // Global environment variables passed to all steps
+  env: {
+    NODE_ENV: 'production',
+  },
+
+  // Custom execution reporters
+  reporters: [
+    new JsonFileReporter({ outputDir: './reports/json' }),
+    new HtmlReporter({ outputDir: './reports/html' }),
+    new SlackReporter({
+      webhookUrl: process.env.SLACK_WEBHOOK_URL,
+      channel: '#ci-deployments',
+    }),
+  ],
+};
 ```
 
-This difference in payload processing allows multiple webhooks from the same source, to separate different events with a similar JSON body.
+---
 
-### Secrets
+## 📐 Deterministic Evaluation Rules
 
-The daemon can fetch secrets from its host environment, or from a file.
+To prevent syntax ambiguity and injection risks, fields in workflow definitions operate under **three strict modes**:
 
-### Inputs
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│ 1. RAW PASSTHROUGH MODE (`run:`)                                         │
+│ • Executed verbatim as a shell process command.                          │
+│ • No string replacements or engine parsing performed.                    │
+│ • Access environment variables strictly via shell syntax: $MY_VAR.       │
+└──────────────────────────────────────────────────────────────────────────┘
+
+┌──────────────────────────────────────────────────────────────────────────┐
+│ 2. EXPRESSION MODE (`if:`, `eval:`)                                      │
+│ • Evaluated strictly as pure JavaScript expressions via Acorn AST.       │
+│ • Must be valid JS syntax (e.g. `inputs.branch === 'main'`).             │
+│ • Automatically coerced to boolean in `if:` conditions.                  │
+└──────────────────────────────────────────────────────────────────────────┘
+
+┌──────────────────────────────────────────────────────────────────────────┐
+│ 3. DETERMINISTIC VALUE MODE (`env:`, `name:`, `image:`, `group:`)        │
+│ • Plain strings WITHOUT `${}` remain 100% raw literal strings.            │
+│ • Strings WITH `${...}` evaluate as standard ES Template Literals.       │
+│ • Example: `node:${inputs.node_version}-alpine`                          │
+└──────────────────────────────────────────────────────────────────────────┘
 
-Inputs are defined from the incoming JSON payload. The payload is parsed and made available as the `inputs` variable
+```
 
-### Mappings
+### Context Scope Available in Expressions
 
-These are shortcuts to make scripting easier.
+Within `${...}`, `if:`, and `eval:` contexts, the following object scopes are exposed:
 
-After the JSON payload is parsed, these mappings are evaluated, and added to `inputs` as shortcuts for long/deep properties in the payload.
+- **`inputs`**: Payload key-values received from incoming webhooks.
+- **`env`**: Merged environment variables from global config and workflow definitions.
+- **`secrets`**: Unmasked secret values loaded from `.env` (`SECRET_` prefix stripped).
+- **`steps`**: Execution statuses and outputs from previous steps in the workflow (`steps.<id>.status`, `steps.<id>.outputs`).
+- **`BUILTIN_HELPERS`**: JS utilities including `String`, `Number`, `Boolean`, and `JSON.parse` / `JSON.stringify`.
 
-For example: from a GitHub webhook event that contains a lot of fields, we can define `image` from `package.package_version.package_url`
+---
 
-```yaml
-mappings:
-  image: ${inputs.package.package_version.package_url}
-```
+## 🌐 Webhook Ingress Gateway & Dashboard
 
-### Environment variables
+The Ingress Gateway listens for incoming HTTP requests and serves the live web UI.
 
-After resolving secrets and mappings, we proceed to resolve env variables from template strings or literal strings.
+### Endpoint Matrix
 
-An env variable is interpreted a JS template string, with `${value}` syntax used to interpolate values from `inputs` or `secrets`.
+| Method     | Endpoint           | Description                                                                            |
+| ---------- | ------------------ | -------------------------------------------------------------------------------------- |
+| **`POST`** | `/webhooks/github` | Webhook endpoint for GitHub events. Evaluates `on.github.if` triggers.                 |
+| **`GET`**  | `/runs`            | **Dashboard:** Live dark-mode monitoring page listing recent jobs and worker health.   |
+| **`GET`**  | `/runs/:jobId`     | **Job Report:** Interactive HTML trace view with step timings and terminal log output. |
 
-## Sequence of operation
+### Dashboard Features
 
-For every incoming event, these steps are followed:
+- **Real-time Auto-Refresh:** `/runs` automatically refreshes job queue statuses (`PENDING`, `RUNNING`, `SUCCESS`, `FAILED`, `CANCELLED`).
+- **ANSI Terminal Rendering:** Uses `ansi_up` to render bash colors, bold highlights, and console outputs accurately in step log boxes.
+- **Payload Inspection:** View JSON inputs received from webhooks for easy debugging.
 
-- Parse and validate payload
-- Load secrets
-- Map inputs
-- Populate env with secrets
-- Populate env with additional workflow definitions (section `env`)
-- Create a temporary working directory
-- Add a volume to defaults at `/workspace`, or a custom path, if a volume with a host path `.` is defined in the workflow
-- Run steps:
-  - For every step, either a string, or a step definition is accepted.
-  - If string, run with the defaults defined in the workflow
-  - If a definition, merge defaults into it, and run the step
-  - The step is a shell command, executed inside a short-lived container
-- Trigger new events
+---
 
-## Triggers
+## 🔒 Security & Hardening
 
-After steps are executed, a list of one or more JSON files can be defined to trigger new workflows.
+1. **Environment Variable Boundary:**
+   By forcing shell steps to consume data via process environment variables (`$CLONE_URL`), malicious webhook payloads containing shell delimiters (e.g. `; rm -rf /`) cannot mutate shell script execution trees.
+2. **Prototype Pollution Protection:**
+   AST evaluation explicitly blocks access to dangerous JS properties (`constructor`, `__proto__`, `prototype`).
+3. **Payload Size Guard:**
+   The Ingress server enforces a strict 5MB payload limit to prevent Out-Of-Memory (OOM) denial-of-service attacks.
+4. **Signal Traps & Resource Cleanup:**
+   Graceful process traps (`SIGINT`, `SIGTERM`) ensure active job handles are safely terminated, file descriptors are closed, and temp `.env`/`.out` files are removed via `try ... finally` blocks.
 
-These files are read one by one and sent back to the daemon as new events.
+## Development
+
+```bash
+pnpm i
+pnpm run lint
+pnpm run test
+pnpm run build
+```

package/dist/config.js

@@ -0,0 +1,15 @@
+/**
+ * Merges user config overrides with baseline defaults
+ */
+export function resolveConfig(userConfig = {}) {
+    return {
+        port: userConfig.port ?? 3000,
+        adminToken: userConfig.adminToken ?? process.env.RUNNER_ADMIN_SECRET ?? '',
+        sqliteUrl: userConfig.sqliteUrl ?? process.env.DATABASE_URL ?? 'sqlite.db',
+        workflowsDir: userConfig.workflowsDir ?? '.on/',
+        workersCount: userConfig.workersCount ?? 5,
+        storagePath: userConfig.storagePath ?? process.env.RUNNER_TMP ?? '/tmp/workspaces',
+        env: userConfig.env ?? {},
+        reporters: userConfig.reporters ?? [],
+    };
+}

package/dist/db-client.js

@@ -0,0 +1,38 @@
+const baseURL = process.env.DATABASE_URL;
+let pragmas = [];
+async function query(method, statement, data, pragma = pragmas) {
+    let req;
+    let error;
+    let retries = 1;
+    let max = 3;
+    while (retries < max) {
+        try {
+            req = await fetch(new URL('/query', baseURL), {
+                method: 'POST',
+                body: JSON.stringify({
+                    s: statement,
+                    d: data,
+                    m: method,
+                    p: pragma,
+                }),
+            });
+            if (req.ok) {
+                return await req.json();
+            }
+            await new Promise((r) => setTimeout(r, retries++ * 1000));
+        }
+        catch (e) {
+            error = e;
+        }
+    }
+    throw new Error(error || (await req.text()));
+}
+export const get = query.bind(null, 'get');
+export const run = query.bind(null, 'run');
+export const all = query.bind(null, 'all');
+export function pragma(p) {
+    if (Array.isArray(p) && p.every((s) => typeof s === 'string')) {
+        pragmas = p;
+    }
+}
+export default { query, get, run, all, pragma };

package/dist/drivers/index.js

@@ -0,0 +1,11 @@
+import { SystemdDriver } from './systemd.driver.js';
+import { StandardProcessDriver } from './standard-process.driver.js';
+export async function resolveDriver() {
+    const systemd = new SystemdDriver();
+    if (await systemd.isSupported()) {
+        console.log('⚡ Selected Execution Driver: Systemd (cgroups enabled)');
+        return systemd;
+    }
+    console.log('📦 Selected Execution Driver: Standard Process (Fallback)');
+    return new StandardProcessDriver();
+}

package/dist/drivers/standard-process.driver.js

@@ -0,0 +1,156 @@
+import { spawn } from 'node:child_process';
+import fs from 'node:fs';
+import path from 'node:path';
+export class StandardProcessDriver {
+    name = 'standard-process';
+    async isSupported() {
+        return true; // Supported on all OS platforms
+    }
+    async execute(ctx) {
+        const startTime = Date.now();
+        let logFd = null;
+        let logFilePath = '';
+        // 1. Guard Log Directory & File Handle Creation
+        try {
+            const logDir = path.join(ctx.workspacePath, '.logs');
+            fs.mkdirSync(logDir, { recursive: true });
+            logFilePath = path.join(logDir, `step-${ctx.stepId}.log`);
+            logFd = fs.openSync(logFilePath, 'a');
+        }
+        catch (err) {
+            return {
+                done: Promise.resolve({
+                    exitCode: 1,
+                    durationMs: 0,
+                    error: new Error(`Failed to initialize step log file: ${err.message}`),
+                }),
+                cancel: async () => { },
+                logFilePath: '',
+            };
+        }
+        // 2. Format Execution Command
+        let cmd;
+        let args;
+        if (ctx.image) {
+            cmd = 'docker';
+            args = [
+                'run',
+                '--rm',
+                '--init',
+                '-v',
+                `${ctx.workspacePath}:/workspace`,
+                '-w',
+                '/workspace',
+                '--entrypoint',
+                'sh',
+                ctx.image,
+                '-c',
+                ctx.command,
+            ];
+        }
+        else {
+            cmd = 'sh';
+            args = ['-c', ctx.command];
+        }
+        // 3. Spawn Detached Child Process
+        let child;
+        try {
+            child = spawn(cmd, args, {
+                cwd: ctx.workspacePath,
+                env: { ...process.env, ...ctx.env },
+                detached: true, // Creates separate Process Group ID
+                stdio: ['ignore', logFd, logFd],
+            });
+        }
+        catch (spawnErr) {
+            try {
+                if (logFd !== null)
+                    fs.closeSync(logFd);
+            }
+            catch { }
+            return {
+                done: Promise.resolve({
+                    exitCode: 1,
+                    durationMs: Date.now() - startTime,
+                    error: new Error(`Failed to spawn process: ${spawnErr.message}`),
+                }),
+                cancel: async () => { },
+                logFilePath,
+            };
+        }
+        let isCancelled = false;
+        let timeoutTimer = null;
+        // 4. Safe Promise Resolution & Lifecycle Tracking
+        const done = new Promise((resolve) => {
+            let isResolved = false;
+            const safeResolve = (result) => {
+                if (isResolved)
+                    return; // Prevent double-resolution
+                isResolved = true;
+                if (timeoutTimer)
+                    clearTimeout(timeoutTimer);
+                // Always close log File Descriptor safely
+                try {
+                    if (logFd !== null)
+                        fs.closeSync(logFd);
+                }
+                catch { }
+                resolve(result);
+            };
+            // Optional step timeout
+            if (ctx.timeoutMs) {
+                timeoutTimer = setTimeout(() => {
+                    isCancelled = true;
+                    this.killProcessGroup(child);
+                }, ctx.timeoutMs);
+                // CRUCIAL: Unref timer so it doesn't hold event loop open
+                timeoutTimer.unref();
+            }
+            child.on('close', (code) => {
+                safeResolve({
+                    exitCode: code ?? (isCancelled ? 130 : 1),
+                    durationMs: Date.now() - startTime,
+                    error: isCancelled ? new Error('Step timed out or was cancelled by user') : undefined,
+                });
+            });
+            child.on('error', (err) => {
+                safeResolve({
+                    exitCode: 1,
+                    durationMs: Date.now() - startTime,
+                    error: err,
+                });
+            });
+        });
+        // 5. Cancellation Hook
+        const cancel = async () => {
+            isCancelled = true;
+            this.killProcessGroup(child);
+        };
+        return { done, cancel, logFilePath };
+    }
+    /**
+     * Kills the entire process group tree (-PID) with unref escalation
+     */
+    killProcessGroup(child) {
+        if (child.pid && !child.killed) {
+            try {
+                // Send SIGTERM to entire process group (-PID)
+                process.kill(-child.pid, 'SIGTERM');
+                // Escalate to SIGKILL after 5 seconds if process tree is still alive
+                const killTimer = setTimeout(() => {
+                    try {
+                        if (child.pid && !child.killed) {
+                            process.kill(-child.pid, 'SIGKILL');
+                        }
+                    }
+                    catch { }
+                }, 5000);
+                // CRUCIAL: Unref escalation timer so Node process can exit cleanly
+                killTimer.unref();
+            }
+            catch {
+                // Process group may already be dead
+            }
+        }
+    }
+}