@livingdata/pipex 0.0.3 → 0.0.5

package/README.md

@@ -4,60 +4,51 @@ Execution engine for containerized steps via Docker CLI.
 
 Runs containers with explicit volume mounts and manages artifacts through a staging/commit lifecycle. Designed to be driven by different orchestrators (CLI included, AI agent planned).
 
-## Installation
+## Prerequisites
+
+- Node.js 24+
+- Docker CLI installed and accessible
+
+## Quick Start
+
+Run directly without installing:
 
 ```bash
-npm install
-cp .env.example .env
-# Edit .env to set PIPEX_WORKDIR if needed (defaults to ./workdir)
+npx @livingdata/pipex run pipeline.yaml
 ```
 
-## Prerequisites
+Or install globally:
 
-- Node.js 24+
-- Docker CLI installed and accessible
+```bash
+npm install -g @livingdata/pipex
+pipex run pipeline.yaml
+```
 
 ## Usage
 
-### Running a pipeline
-
 ```bash
 # Interactive mode (default)
-npm start -- run pipeline.example.json
-
-# With workspace name (enables caching)
-npm start -- run pipeline.example.json --workspace my-build
+pipex run pipeline.yaml
 
 # JSON mode (for CI/CD)
-npm start -- run pipeline.example.json --json
+pipex run pipeline.yaml --json
 
 # Custom workdir
-npm start -- run pipeline.example.json --workdir /tmp/builds
+pipex run pipeline.yaml --workdir /tmp/builds
 ```
 
 ### Managing workspaces
 
 ```bash
 # List workspaces (with artifact/cache counts)
-npm start -- list
-npm start -- ls --json
+pipex list
+pipex ls --json
 
 # Remove specific workspaces
-npm start -- rm my-build other-build
+pipex rm my-build other-build
 
 # Remove all workspaces
-npm start -- clean
-```
-
-### Via npx
-
-```bash
-# Build first
-npm run build
-
-# Run locally via npx
-npx . run example/pipeline.json --workspace my-build
-npx . list
+pipex clean
 ```
 
 ### Commands
@@ -85,34 +76,138 @@ npx . list
 
 ## Pipeline Format
 
-Minimal example:
-
-```json
-{
-  "name": "my-pipeline",
-  "steps": [
-    {
-      "id": "download",
-      "image": "alpine:3.19",
-      "cmd": ["sh", "-c", "echo hello > /output/hello.txt"]
-    },
-    {
-      "id": "process",
-      "image": "alpine:3.19",
-      "cmd": ["cat", "/input/download/hello.txt"],
-      "inputs": [{"step": "download"}]
-    }
-  ]
-}
+Pipeline files can be written in **YAML** (`.yaml` / `.yml`) or **JSON** (`.json`). YAML is recommended for readability; JSON is still fully supported.
+
+Steps can be defined in two ways: **raw steps** with explicit image/cmd, or **kit steps** using `uses` for common patterns. Both can coexist in the same pipeline.
+
+### Pipeline and Step Identity
+
+Both pipelines and steps support an `id`/`name` duality:
+
+- **`id`** — Machine identifier (alphanum, dash, underscore). Used for caching, state, artifacts.
+- **`name`** — Human-readable label (free-form text). Used for display.
+- At least one must be defined. If `id` is missing it is derived from `name` via slugification (e.g. `"Données préparées"` → `donnees-preparees`). If `name` is missing, `id` is used for display.
+
+```yaml
+# Pipeline with both id and name
+id: data-pipeline
+name: Data Processing Pipeline
+steps:
+  # Step with only id (current style, still works)
+  - id: download
+    image: alpine:3.19
+    cmd: [sh, -c, "echo hello > /output/hello.txt"]
+
+  # Step with only name (id auto-derived to "build-assets")
+  - name: Build Assets
+    image: node:22-alpine
+    cmd: [sh, -c, "echo done > /output/result.txt"]
+
+  # Step with both
+  - id: deploy
+    name: Deploy to Staging
+    image: alpine:3.19
+    cmd: [echo, deployed]
+```
+
+### Kit Steps
+
+Kits are reusable templates that generate the image, command, caches, and mounts for common runtimes. Use `uses` to select a kit and `with` to pass parameters:
+
+```yaml
+name: my-pipeline
+steps:
+  - id: build
+    uses: node
+    with: { script: build.js, src: src/app }
+  - id: analyze
+    uses: python
+    with: { script: analyze.py, src: scripts }
+  - id: extract
+    uses: shell
+    with: { packages: [unzip], run: "unzip /input/build/archive.zip -d /output/" }
+    inputs: [{ step: build }]
+```
+
+`uses` and `image`/`cmd` are mutually exclusive. All other step fields (`env`, `inputs`, `mounts`, `caches`, `timeoutSec`, `allowFailure`, `allowNetwork`) remain available and merge with kit defaults (user values take priority). The `src` parameter in `with` generates a read-only mount at `/app` in the container.
+
+#### Available Kits
+
+**`node`** -- Run a Node.js script with automatic dependency installation.
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `script` | *(required)* | Script to run (relative to `/app`) |
+| `src` | -- | Host directory to mount at `/app` |
+| `version` | `"24"` | Node.js version |
+| `packageManager` | `"npm"` | `"npm"`, `"pnpm"`, or `"yarn"` |
+| `install` | `true` | Run package install before script |
+| `variant` | `"alpine"` | Image variant |
+
+**`python`** -- Run a Python script with automatic dependency installation from `requirements.txt`.
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `script` | *(required)* | Script to run (relative to `/app`) |
+| `src` | -- | Host directory to mount at `/app` |
+| `version` | `"3.12"` | Python version |
+| `packageManager` | `"pip"` | `"pip"` or `"uv"` |
+| `install` | `true` | Run dependency install before script |
+| `variant` | `"slim"` | Image variant |
+
+**`shell`** -- Run a shell command in a container, with optional apt package installation.
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `run` | *(required)* | Shell command to execute |
+| `packages` | -- | Apt packages to install before running |
+| `src` | -- | Host directory to mount at `/app` |
+| `image` | `"alpine:3.20"` | Docker image (defaults to `"debian:bookworm-slim"` when `packages` is set) |
+
+When `packages` is provided, the kit automatically switches to a Debian image, enables network access, and provides an `apt-cache` cache. Without packages, it runs on a minimal Alpine image with no network.
+
+```yaml
+# Simple command (alpine, no network)
+- id: list-files
+  uses: shell
+  with:
+    run: ls -lhR /input/data/
+
+# With system packages (debian, network + apt cache)
+- id: extract
+  uses: shell
+  with:
+    packages: [unzip, jq]
+    run: unzip /input/download/data.zip -d /output/
+  inputs: [{ step: download }]
+```
+
+### Raw Steps
+
+For full control, define `image` and `cmd` directly:
+
+```yaml
+name: my-pipeline
+steps:
+  - id: download
+    image: alpine:3.19
+    cmd: [sh, -c, "echo hello > /output/hello.txt"]
+  - id: process
+    image: alpine:3.19
+    cmd: [cat, /input/download/hello.txt]
+    inputs: [{ step: download }]
 ```
 
 ### Step Options
 
 | Field | Type | Description |
 |-------|------|-------------|
-| `id` | string | Step identifier (required) |
-| `image` | string | Docker image (required) |
-| `cmd` | string[] | Command to execute (required) |
+| `id` | string | Step identifier (at least one of `id`/`name` required) |
+| `name` | string | Human-readable display name |
+| `image` | string | Docker image (required for raw steps) |
+| `cmd` | string[] | Command to execute (required for raw steps) |
+| `uses` | string | Kit name (required for kit steps) |
+| `with` | object | Kit parameters |
 | `inputs` | InputSpec[] | Previous steps to mount as read-only |
 | `env` | Record<string, string> | Environment variables |
 | `outputPath` | string | Output mount point (default: `/output`) |
@@ -126,11 +221,11 @@ Minimal example:
 
 Mount previous steps as read-only:
 
-```json
-"inputs": [
-  {"step": "step1"},
-  {"step": "step2", "copyToOutput": true}
-]
+```yaml
+inputs:
+  - step: step1
+  - step: step2
+    copyToOutput: true
 ```
 
 - Mounted under `/input/{stepName}/`
@@ -140,11 +235,12 @@ Mount previous steps as read-only:
 
 Mount host directories into containers as **read-only**:
 
-```json
-"mounts": [
-  {"host": "src/app", "container": "/app"},
-  {"host": "config", "container": "/config"}
-]
+```yaml
+mounts:
+  - host: src/app
+    container: /app
+  - host: config
+    container: /config
 ```
 
 - `host` must be a **relative** path (resolved from the pipeline file's directory)
@@ -152,17 +248,18 @@ Mount host directories into containers as **read-only**:
 - Neither path can contain `..`
 - Always mounted read-only -- containers cannot modify host files
 
-This means a pipeline at `/project/ci/pipeline.json` can only mount subdirectories of `/project/ci/`. Use `/tmp` or `/output` inside the container for writes.
+This means a pipeline at `/project/ci/pipeline.yaml` can only mount subdirectories of `/project/ci/`. Use `/tmp` or `/output` inside the container for writes.
 
 ### Caches
 
 Persistent read-write directories shared across steps and executions:
 
-```json
-"caches": [
-  {"name": "pnpm-store", "path": "/root/.local/share/pnpm/store"},
-  {"name": "build-cache", "path": "/tmp/cache"}
-]
+```yaml
+caches:
+  - name: pnpm-store
+    path: /root/.local/share/pnpm/store
+  - name: build-cache
+    path: /tmp/cache
 ```
 
 - **Persistent**: Caches survive across pipeline executions
@@ -176,37 +273,53 @@ Common use cases:
 
 **Note**: Caches are workspace-scoped (not global). Different workspaces have isolated caches.
 
-## Example
+## Examples
 
-The `example/` directory contains a multi-language pipeline that chains Node.js and Python steps:
+### Geodata Processing
 
+The `examples/geodata/` pipeline downloads a shapefile archive, extracts it, and produces a CSV inventory — using the `debian` and `bash` kits:
+
+```
+examples/geodata/
+└── pipeline.yaml
 ```
-example/
-├── pipeline.json
+
+Steps: `download` → `extract` → `list-files` / `build-csv`
+
+```bash
+pipex run examples/geodata/pipeline.yaml
+```
+
+### Multi-Language
+
+The `examples/multi-language/` pipeline chains Node.js and Python steps using kits:
+
+```
+examples/multi-language/
+├── pipeline.yaml
 └── scripts/
-    ├── nodejs/          # lodash-based data analysis
+    ├── nodejs/              # lodash-based data analysis
     │   ├── package.json
     │   ├── analyze.js
     │   └── transform.js
-    └── python/          # pyyaml-based enrichment
+    └── python/              # pyyaml-based enrichment
         ├── pyproject.toml
+        ├── requirements.txt
         ├── analyze.py
         └── transform.py
 ```
 
-The pipeline runs 4 steps: `node-analyze` → `node-transform` → `python-analyze` → `python-transform`. Each step mounts its scripts directory as read-only and passes artifacts to the next step via `/input`.
+Steps: `node-analyze` → `node-transform` → `python-analyze` → `python-transform`
 
 ```bash
-npm start -- run example/pipeline.json --workspace example-test
+pipex run examples/multi-language/pipeline.yaml
 ```
 
 ## Caching & Workspaces
 
-Workspaces enable caching across runs. Name is determined by:
+Workspaces enable caching across runs. The workspace ID is determined by:
 1. CLI flag `--workspace` (highest priority)
-2. Config `"name"` field
-3. Filename (e.g., `build.json` → `build`)
-4. Auto-generated timestamp
+2. Pipeline `id` (explicit or derived from `name`)
 
 **Cache behavior**: Steps are skipped if image, cmd, env, inputs, and mounts haven't changed. See code documentation for details.
 
@@ -232,10 +345,10 @@ newgrp docker
 Clean old workspaces:
 
 ```bash
-npm start -- list
-npm start -- rm old-workspace-id
+pipex list
+pipex rm old-workspace-id
 # Or remove all at once
-npm start -- clean
+pipex clean
 ```
 
 ### Cached step with missing artifact
@@ -249,9 +362,25 @@ rm $PIPEX_WORKDIR/{workspace-id}/state.json
 ## Development
 
 ```bash
-npm run build
-npm run lint
-npm run lint:fix
+git clone https://github.com/livingdata-co/pipex.git
+cd pipex
+npm install
+cp .env.example .env
+```
+
+Run the CLI without building (via tsx):
+
+```bash
+npm run cli -- run pipeline.yaml
+npm run cli -- list
+```
+
+Other commands:
+
+```bash
+npm run build        # Compile TypeScript (tsc → dist/)
+npm run lint         # Lint with XO
+npm run lint:fix     # Auto-fix lint issues
 ```
 
 ## Architecture
@@ -259,3 +388,4 @@ npm run lint:fix
 For implementation details, see code documentation in:
 - `src/engine/` - Low-level container execution (workspace, executor)
 - `src/cli/` - Pipeline orchestration (runner, loader, state)
+- `src/kits/` - Kit system (registry, built-in kit implementations)

package/dist/cli/index.js

@@ -23,7 +23,7 @@ async function main() {
     program
         .command('run')
         .description('Execute a pipeline')
-        .argument('<pipeline>', 'Pipeline JSON file to execute')
+        .argument('<pipeline>', 'Pipeline file to execute (JSON or YAML)')
         .option('-w, --workspace <name>', 'Workspace name (for caching)')
         .option('-f, --force [steps]', 'Skip cache for all steps, or a comma-separated list (e.g. --force step1,step2)')
         .action(async (pipelineFile, options, cmd) => {

package/dist/cli/pipeline-loader.js

@@ -1,20 +1,60 @@
 import { readFile } from 'node:fs/promises';
+import { extname } from 'node:path';
+import { deburr } from 'lodash-es';
+import { parse as parseYaml } from 'yaml';
+import { getKit } from '../kits/index.js';
+import { isKitStep } from '../types.js';
 export class PipelineLoader {
     async load(filePath) {
         const content = await readFile(filePath, 'utf8');
-        const config = JSON.parse(content);
-        if (!Array.isArray(config.steps) || config.steps.length === 0) {
+        return this.parse(content, filePath);
+    }
+    parse(content, filePath) {
+        const input = parsePipelineFile(content, filePath);
+        if (!input.id && !input.name) {
+            throw new Error('Invalid pipeline: at least one of "id" or "name" must be defined');
+        }
+        const pipelineId = input.id ?? slugify(input.name);
+        if (!Array.isArray(input.steps) || input.steps.length === 0) {
             throw new Error('Invalid pipeline: steps must be a non-empty array');
         }
-        for (const step of config.steps) {
+        const steps = input.steps.map(step => this.resolveStep(step));
+        for (const step of steps) {
             this.validateStep(step);
         }
-        return config;
+        this.validateUniqueStepIds(steps);
+        return { id: pipelineId, name: input.name, steps };
     }
-    validateStep(step) {
-        if (!step.id || typeof step.id !== 'string') {
-            throw new Error('Invalid step: id is required');
+    resolveStep(step) {
+        if (!step.id && !step.name) {
+            throw new Error('Invalid step: at least one of "id" or "name" must be defined');
         }
+        const id = step.id ?? slugify(step.name);
+        const { name } = step;
+        if (!isKitStep(step)) {
+            return { ...step, id, name };
+        }
+        return this.resolveKitStep(step, id, name);
+    }
+    resolveKitStep(step, id, name) {
+        const kit = getKit(step.uses);
+        const kitOutput = kit.resolve(step.with ?? {});
+        return {
+            id,
+            name,
+            image: kitOutput.image,
+            cmd: kitOutput.cmd,
+            env: mergeEnv(kitOutput.env, step.env),
+            inputs: step.inputs,
+            outputPath: step.outputPath,
+            caches: mergeCaches(kitOutput.caches, step.caches),
+            mounts: mergeMounts(kitOutput.mounts, step.mounts),
+            timeoutSec: step.timeoutSec,
+            allowFailure: step.allowFailure,
+            allowNetwork: step.allowNetwork ?? kitOutput.allowNetwork
+        };
+    }
+    validateStep(step) {
         this.validateIdentifier(step.id, 'step id');
         if (!step.image || typeof step.image !== 'string') {
             throw new Error(`Invalid step ${step.id}: image is required`);
@@ -84,4 +124,54 @@ export class PipelineLoader {
             throw new Error(`Invalid ${context}: '${id}' cannot contain '..'`);
         }
     }
+    validateUniqueStepIds(steps) {
+        const seen = new Set();
+        for (const step of steps) {
+            if (seen.has(step.id)) {
+                throw new Error(`Duplicate step id: '${step.id}'`);
+            }
+            seen.add(step.id);
+        }
+    }
+}
+/** Convert a free-form name into a valid identifier. */
+export function slugify(name) {
+    return deburr(name)
+        .toLowerCase()
+        .replaceAll(/[^\w-]/g, '-')
+        .replaceAll(/-{2,}/g, '-')
+        .replace(/^-/, '')
+        .replace(/-$/, '');
+}
+export function parsePipelineFile(content, filePath) {
+    const ext = extname(filePath).toLowerCase();
+    if (ext === '.yaml' || ext === '.yml') {
+        return parseYaml(content);
+    }
+    return JSON.parse(content);
+}
+export function mergeEnv(kitEnv, userEnv) {
+    if (!kitEnv && !userEnv) {
+        return undefined;
+    }
+    return { ...kitEnv, ...userEnv };
+}
+export function mergeCaches(kitCaches, userCaches) {
+    if (!kitCaches && !userCaches) {
+        return undefined;
+    }
+    const map = new Map();
+    for (const c of kitCaches ?? []) {
+        map.set(c.name, c);
+    }
+    for (const c of userCaches ?? []) {
+        map.set(c.name, c);
+    }
+    return [...map.values()];
+}
+export function mergeMounts(kitMounts, userMounts) {
+    if (!kitMounts && !userMounts) {
+        return undefined;
+    }
+    return [...(kitMounts ?? []), ...(userMounts ?? [])];
 }

package/dist/cli/pipeline-runner.js

@@ -1,5 +1,5 @@
 import { cp } from 'node:fs/promises';
-import { basename, dirname, resolve } from 'node:path';
+import { dirname, resolve } from 'node:path';
 import { Workspace } from '../engine/index.js';
 import { StateManager } from './state.js';
 /**
@@ -47,10 +47,8 @@ export class PipelineRunner {
         const { workspace: workspaceName, force } = options ?? {};
         const config = await this.loader.load(pipelineFilePath);
         const pipelineRoot = dirname(resolve(pipelineFilePath));
-        // Workspace ID priority: CLI arg > config.name > filename
-        const workspaceId = workspaceName
-            ?? config.name
-            ?? basename(pipelineFilePath, '.json').replaceAll(/[^\w-]/g, '-');
+        // Workspace ID priority: CLI arg > pipeline id
+        const workspaceId = workspaceName ?? config.id;
         let workspace;
         try {
             workspace = await Workspace.open(this.workdirRoot, workspaceId);
@@ -63,8 +61,9 @@ export class PipelineRunner {
         const state = new StateManager(workspace.root);
         await state.load();
         const stepArtifacts = new Map();
-        this.reporter.state(workspace.id, 'PIPELINE_START');
+        this.reporter.state(workspace.id, 'PIPELINE_START', undefined, { pipelineName: config.name ?? config.id });
         for (const step of config.steps) {
+            const stepRef = { id: step.id, displayName: step.name ?? step.id };
             const inputArtifactIds = step.inputs
                 ?.map(i => stepArtifacts.get(i.step))
                 .filter((id) => id !== undefined);
@@ -80,10 +79,10 @@ export class PipelineRunner {
                 mounts: resolvedMounts
             });
             const skipCache = force === true || (Array.isArray(force) && force.includes(step.id));
-            if (!skipCache && await this.tryUseCache({ workspace, state, step, currentFingerprint, stepArtifacts })) {
+            if (!skipCache && await this.tryUseCache({ workspace, state, step, stepRef, currentFingerprint, stepArtifacts })) {
                 continue;
             }
-            this.reporter.state(workspace.id, 'STEP_STARTING', step.id);
+            this.reporter.state(workspace.id, 'STEP_STARTING', stepRef);
             const artifactId = workspace.generateArtifactId();
             const stagingPath = await workspace.prepareArtifact(artifactId);
             await this.prepareStagingWithInputs(workspace, step, stagingPath, stepArtifacts);
@@ -106,33 +105,33 @@ export class PipelineRunner {
                 network: step.allowNetwork ? 'bridge' : 'none',
                 timeoutSec: step.timeoutSec
             }, ({ stream, line }) => {
-                this.reporter.log(workspace.id, step.id, stream, line);
+                this.reporter.log(workspace.id, stepRef, stream, line);
             });
-            this.reporter.result(workspace.id, step.id, result);
+            this.reporter.result(workspace.id, stepRef, result);
             if (result.exitCode === 0 || step.allowFailure) {
                 await workspace.commitArtifact(artifactId);
                 stepArtifacts.set(step.id, artifactId);
                 state.setStep(step.id, artifactId, currentFingerprint);
                 await state.save();
-                this.reporter.state(workspace.id, 'STEP_FINISHED', step.id, { artifactId });
+                this.reporter.state(workspace.id, 'STEP_FINISHED', stepRef, { artifactId });
             }
             else {
                 await workspace.discardArtifact(artifactId);
-                this.reporter.state(workspace.id, 'STEP_FAILED', step.id, { exitCode: result.exitCode });
+                this.reporter.state(workspace.id, 'STEP_FAILED', stepRef, { exitCode: result.exitCode });
                 this.reporter.state(workspace.id, 'PIPELINE_FAILED');
                 throw new Error(`Step ${step.id} failed with exit code ${result.exitCode}`);
             }
         }
         this.reporter.state(workspace.id, 'PIPELINE_FINISHED');
     }
-    async tryUseCache({ workspace, state, step, currentFingerprint, stepArtifacts }) {
+    async tryUseCache({ workspace, state, step, stepRef, currentFingerprint, stepArtifacts }) {
         const cached = state.getStep(step.id);
         if (cached?.fingerprint === currentFingerprint) {
             try {
                 const artifacts = await workspace.listArtifacts();
                 if (artifacts.includes(cached.artifactId)) {
                     stepArtifacts.set(step.id, cached.artifactId);
-                    this.reporter.state(workspace.id, 'STEP_SKIPPED', step.id, { artifactId: cached.artifactId, reason: 'cached' });
+                    this.reporter.state(workspace.id, 'STEP_SKIPPED', stepRef, { artifactId: cached.artifactId, reason: 'cached' });
                     return true;
                 }
             }

package/dist/cli/reporter.js

@@ -7,14 +7,15 @@ import ora from 'ora';
  */
 export class ConsoleReporter {
     logger = pino({ level: 'info' });
-    state(workspaceId, event, stepId, meta) {
-        this.logger.info({ workspaceId, event, stepId, ...meta });
+    state(workspaceId, event, step, meta) {
+        const stepName = step?.displayName === step?.id ? undefined : step?.displayName;
+        this.logger.info({ workspaceId, event, stepId: step?.id, stepName, ...meta });
     }
-    log(workspaceId, stepId, stream, line) {
-        this.logger.info({ workspaceId, stepId, stream, line });
+    log(workspaceId, step, stream, line) {
+        this.logger.info({ workspaceId, stepId: step.id, stream, line });
     }
-    result(workspaceId, stepId, result) {
-        this.logger.info({ workspaceId, stepId, result });
+    result(workspaceId, step, result) {
+        this.logger.info({ workspaceId, stepId: step.id, result });
     }
 }
 /**
@@ -24,42 +25,42 @@ export class ConsoleReporter {
 export class InteractiveReporter {
     spinner;
     stepSpinners = new Map();
-    state(workspaceId, event, stepId, meta) {
+    state(workspaceId, event, step, meta) {
         if (event === 'PIPELINE_START') {
-            console.log(chalk.bold(`\n▶ Pipeline: ${chalk.cyan(workspaceId)}\n`));
+            const displayName = meta?.pipelineName ?? workspaceId;
+            console.log(chalk.bold(`\n▶ Pipeline: ${chalk.cyan(displayName)}\n`));
         }
-        if (event === 'STEP_STARTING' && stepId) {
-            const spinner = ora({ text: stepId, prefixText: '  ' }).start();
-            this.stepSpinners.set(stepId, spinner);
+        if (event === 'STEP_STARTING' && step) {
+            const spinner = ora({ text: step.displayName, prefixText: '  ' }).start();
+            this.stepSpinners.set(step.id, spinner);
         }
-        if (event === 'STEP_SKIPPED' && stepId) {
-            const spinner = this.stepSpinners.get(stepId);
+        if (event === 'STEP_SKIPPED' && step) {
+            const spinner = this.stepSpinners.get(step.id);
             if (spinner) {
-                spinner.stopAndPersist({ symbol: chalk.gray('⊙'), text: chalk.gray(`${stepId} (cached)`) });
-                this.stepSpinners.delete(stepId);
+                spinner.stopAndPersist({ symbol: chalk.gray('⊙'), text: chalk.gray(`${step.displayName} (cached)`) });
+                this.stepSpinners.delete(step.id);
             }
             else {
-                // Step was skipped before spinner was created
-                console.log(`  ${chalk.gray('⊙')} ${chalk.gray(`${stepId} (cached)`)}`);
+                console.log(`  ${chalk.gray('⊙')} ${chalk.gray(`${step.displayName} (cached)`)}`);
             }
         }
-        if (event === 'STEP_FINISHED' && stepId) {
-            const spinner = this.stepSpinners.get(stepId);
+        if (event === 'STEP_FINISHED' && step) {
+            const spinner = this.stepSpinners.get(step.id);
             if (spinner) {
-                spinner.stopAndPersist({ symbol: chalk.green('✓'), text: chalk.green(stepId) });
-                this.stepSpinners.delete(stepId);
+                spinner.stopAndPersist({ symbol: chalk.green('✓'), text: chalk.green(step.displayName) });
+                this.stepSpinners.delete(step.id);
             }
         }
-        if (event === 'STEP_FAILED' && stepId) {
-            const spinner = this.stepSpinners.get(stepId);
+        if (event === 'STEP_FAILED' && step) {
+            const spinner = this.stepSpinners.get(step.id);
             const exitCode = meta?.exitCode;
             if (spinner) {
                 const exitInfo = exitCode === undefined ? '' : ` (exit ${exitCode})`;
                 spinner.stopAndPersist({
                     symbol: chalk.red('✗'),
-                    text: chalk.red(`${stepId}${exitInfo}`)
+                    text: chalk.red(`${step.displayName}${exitInfo}`)
                 });
-                this.stepSpinners.delete(stepId);
+                this.stepSpinners.delete(step.id);
             }
         }
         if (event === 'PIPELINE_FINISHED') {
@@ -69,10 +70,10 @@ export class InteractiveReporter {
             console.log(chalk.bold.red('\n✗ Pipeline failed\n'));
         }
     }
-    log(_workspaceId, _stepId, _stream, _line) {
+    log(_workspaceId, _step, _stream, _line) {
         // Suppress logs in interactive mode
     }
-    result(_workspaceId, _stepId, _result) {
+    result(_workspaceId, _step, _result) {
         // Results shown via state updates
     }
 }

package/dist/cli/types.js

@@ -1 +1,3 @@
-export {};
+export function isKitStep(step) {
+    return 'uses' in step && typeof step.uses === 'string';
+}

package/dist/kits/bash.js

@@ -0,0 +1,19 @@
+export const bashKit = {
+    name: 'bash',
+    resolve(params) {
+        const run = params.run;
+        if (!run || typeof run !== 'string') {
+            throw new Error('Kit "bash": "run" parameter is required');
+        }
+        const image = params.image ?? 'alpine:3.20';
+        const src = params.src;
+        const output = {
+            image,
+            cmd: ['sh', '-c', run]
+        };
+        if (src) {
+            output.mounts = [{ host: src, container: '/app' }];
+        }
+        return output;
+    }
+};

package/dist/kits/builtin/bash.js

@@ -0,0 +1,19 @@
+export const bashKit = {
+    name: 'bash',
+    resolve(params) {
+        const run = params.run;
+        if (!run || typeof run !== 'string') {
+            throw new Error('Kit "bash": "run" parameter is required');
+        }
+        const image = params.image ?? 'alpine:3.20';
+        const src = params.src;
+        const output = {
+            image,
+            cmd: ['sh', '-c', run]
+        };
+        if (src) {
+            output.mounts = [{ host: src, container: '/app' }];
+        }
+        return output;
+    }
+};

package/dist/kits/builtin/node.js

@@ -0,0 +1,56 @@
+const cacheMap = {
+    npm: { name: 'npm-cache', path: '/root/.npm' },
+    pnpm: { name: 'pnpm-store', path: '/root/.local/share/pnpm/store' },
+    yarn: { name: 'yarn-cache', path: '/usr/local/share/.cache/yarn' }
+};
+function buildInstallCommand(packageManager) {
+    switch (packageManager) {
+        case 'npm': {
+            return 'cd /tmp && cp /app/package*.json . && npm install --no-audit --no-fund 2>&1';
+        }
+        case 'pnpm': {
+            return 'cd /tmp && cp /app/package.json . && cp /app/pnpm-lock.yaml . 2>/dev/null; pnpm install --no-frozen-lockfile 2>&1';
+        }
+        case 'yarn': {
+            return 'cd /tmp && cp /app/package.json . && cp /app/yarn.lock . 2>/dev/null; yarn install 2>&1';
+        }
+        default: {
+            throw new Error(`Kit "node": unsupported packageManager "${packageManager}"`);
+        }
+    }
+}
+export const nodeKit = {
+    name: 'node',
+    resolve(params) {
+        const version = params.version ?? '24';
+        const packageManager = params.packageManager ?? 'npm';
+        const script = params.script;
+        const install = params.install ?? true;
+        const variant = params.variant ?? 'alpine';
+        const src = params.src;
+        if (!script || typeof script !== 'string') {
+            throw new Error('Kit "node": "script" parameter is required');
+        }
+        const image = `node:${version}-${variant}`;
+        const parts = [];
+        if (install) {
+            parts.push(buildInstallCommand(packageManager));
+        }
+        const nodePathPrefix = install ? 'NODE_PATH=/tmp/node_modules ' : '';
+        parts.push(`${nodePathPrefix}node /app/${script}`);
+        const cache = cacheMap[packageManager];
+        if (!cache) {
+            throw new Error(`Kit "node": unsupported packageManager "${packageManager}"`);
+        }
+        const output = {
+            image,
+            cmd: ['sh', '-c', parts.join(' && ')],
+            caches: [cache],
+            allowNetwork: true
+        };
+        if (src) {
+            output.mounts = [{ host: src, container: '/app' }];
+        }
+        return output;
+    }
+};

package/dist/kits/builtin/python.js

@@ -0,0 +1,51 @@
+const cacheMap = {
+    pip: { name: 'pip-cache', path: '/root/.cache/pip' },
+    uv: { name: 'uv-cache', path: '/root/.cache/uv' }
+};
+function buildInstallCommand(packageManager) {
+    switch (packageManager) {
+        case 'pip': {
+            return 'pip install --quiet -r /app/requirements.txt 2>&1';
+        }
+        case 'uv': {
+            return 'uv pip install --quiet -r /app/requirements.txt 2>&1';
+        }
+        default: {
+            throw new Error(`Kit "python": unsupported packageManager "${packageManager}"`);
+        }
+    }
+}
+export const pythonKit = {
+    name: 'python',
+    resolve(params) {
+        const version = params.version ?? '3.12';
+        const packageManager = params.packageManager ?? 'pip';
+        const script = params.script;
+        const install = params.install ?? true;
+        const variant = params.variant ?? 'slim';
+        const src = params.src;
+        if (!script || typeof script !== 'string') {
+            throw new Error('Kit "python": "script" parameter is required');
+        }
+        const image = `python:${version}-${variant}`;
+        const cache = cacheMap[packageManager];
+        if (!cache) {
+            throw new Error(`Kit "python": unsupported packageManager "${packageManager}"`);
+        }
+        const parts = [];
+        if (install) {
+            parts.push(buildInstallCommand(packageManager));
+        }
+        parts.push(`python /app/${script}`);
+        const output = {
+            image,
+            cmd: ['sh', '-c', parts.join(' && ')],
+            caches: [cache],
+            allowNetwork: true
+        };
+        if (src) {
+            output.mounts = [{ host: src, container: '/app' }];
+        }
+        return output;
+    }
+};

package/dist/kits/builtin/shell.js

@@ -0,0 +1,31 @@
+export const shellKit = {
+    name: 'shell',
+    resolve(params) {
+        const run = params.run;
+        if (!run || typeof run !== 'string') {
+            throw new Error('Kit "shell": "run" parameter is required');
+        }
+        const packages = params.packages;
+        const hasPackages = packages && packages.length > 0;
+        const src = params.src;
+        const defaultImage = hasPackages ? 'debian:bookworm-slim' : 'alpine:3.20';
+        const image = params.image ?? defaultImage;
+        const parts = [];
+        if (hasPackages) {
+            parts.push(`apt-get update && apt-get install -y --no-install-recommends ${packages.join(' ')} && rm -rf /var/lib/apt/lists/*`);
+        }
+        parts.push(run);
+        const output = {
+            image,
+            cmd: ['sh', '-c', parts.join(' && ')]
+        };
+        if (hasPackages) {
+            output.caches = [{ name: 'apt-cache', path: '/var/cache/apt' }];
+            output.allowNetwork = true;
+        }
+        if (src) {
+            output.mounts = [{ host: src, container: '/app' }];
+        }
+        return output;
+    }
+};

package/dist/kits/index.js

@@ -0,0 +1,15 @@
+import { nodeKit } from './builtin/node.js';
+import { pythonKit } from './builtin/python.js';
+import { shellKit } from './builtin/shell.js';
+const kits = new Map([
+    [nodeKit.name, nodeKit],
+    [pythonKit.name, pythonKit],
+    [shellKit.name, shellKit]
+]);
+export function getKit(name) {
+    const kit = kits.get(name);
+    if (!kit) {
+        throw new Error(`Unknown kit: "${name}". Available kits: ${[...kits.keys()].join(', ')}`);
+    }
+    return kit;
+}

package/dist/kits/node.js

@@ -0,0 +1,56 @@
+const cacheMap = {
+    npm: { name: 'npm-cache', path: '/root/.npm' },
+    pnpm: { name: 'pnpm-store', path: '/root/.local/share/pnpm/store' },
+    yarn: { name: 'yarn-cache', path: '/usr/local/share/.cache/yarn' }
+};
+function buildInstallCommand(packageManager) {
+    switch (packageManager) {
+        case 'npm': {
+            return 'cd /tmp && cp /app/package*.json . && npm install --no-audit --no-fund 2>&1';
+        }
+        case 'pnpm': {
+            return 'cd /tmp && cp /app/package.json . && cp /app/pnpm-lock.yaml . 2>/dev/null; pnpm install --no-frozen-lockfile 2>&1';
+        }
+        case 'yarn': {
+            return 'cd /tmp && cp /app/package.json . && cp /app/yarn.lock . 2>/dev/null; yarn install 2>&1';
+        }
+        default: {
+            throw new Error(`Kit "node": unsupported packageManager "${packageManager}"`);
+        }
+    }
+}
+export const nodeKit = {
+    name: 'node',
+    resolve(params) {
+        const version = params.version ?? '22';
+        const packageManager = params.packageManager ?? 'npm';
+        const script = params.script;
+        const install = params.install ?? true;
+        const variant = params.variant ?? 'alpine';
+        const src = params.src;
+        if (!script || typeof script !== 'string') {
+            throw new Error('Kit "node": "script" parameter is required');
+        }
+        const image = `node:${version}-${variant}`;
+        const parts = [];
+        if (install) {
+            parts.push(buildInstallCommand(packageManager));
+        }
+        const nodePathPrefix = install ? 'NODE_PATH=/tmp/node_modules ' : '';
+        parts.push(`${nodePathPrefix}node /app/${script}`);
+        const cache = cacheMap[packageManager];
+        if (!cache) {
+            throw new Error(`Kit "node": unsupported packageManager "${packageManager}"`);
+        }
+        const output = {
+            image,
+            cmd: ['sh', '-c', parts.join(' && ')],
+            caches: [cache],
+            allowNetwork: true
+        };
+        if (src) {
+            output.mounts = [{ host: src, container: '/app' }];
+        }
+        return output;
+    }
+};

package/dist/kits/python.js

@@ -0,0 +1,51 @@
+const cacheMap = {
+    pip: { name: 'pip-cache', path: '/root/.cache/pip' },
+    uv: { name: 'uv-cache', path: '/root/.cache/uv' }
+};
+function buildInstallCommand(packageManager) {
+    switch (packageManager) {
+        case 'pip': {
+            return 'pip install --quiet /app/ 2>&1';
+        }
+        case 'uv': {
+            return 'uv pip install --quiet /app/ 2>&1';
+        }
+        default: {
+            throw new Error(`Kit "python": unsupported packageManager "${packageManager}"`);
+        }
+    }
+}
+export const pythonKit = {
+    name: 'python',
+    resolve(params) {
+        const version = params.version ?? '3.12';
+        const packageManager = params.packageManager ?? 'pip';
+        const script = params.script;
+        const install = params.install ?? true;
+        const variant = params.variant ?? 'slim';
+        const src = params.src;
+        if (!script || typeof script !== 'string') {
+            throw new Error('Kit "python": "script" parameter is required');
+        }
+        const image = `python:${version}-${variant}`;
+        const cache = cacheMap[packageManager];
+        if (!cache) {
+            throw new Error(`Kit "python": unsupported packageManager "${packageManager}"`);
+        }
+        const parts = [];
+        if (install) {
+            parts.push(buildInstallCommand(packageManager));
+        }
+        parts.push(`python /app/${script}`);
+        const output = {
+            image,
+            cmd: ['sh', '-c', parts.join(' && ')],
+            caches: [cache],
+            allowNetwork: true
+        };
+        if (src) {
+            output.mounts = [{ host: src, container: '/app' }];
+        }
+        return output;
+    }
+};

package/dist/kits/types.js

@@ -0,0 +1 @@
+export {};

package/dist/types.js

@@ -0,0 +1,10 @@
+// ---------------------------------------------------------------------------
+// Shared pipeline domain types.
+//
+// These types are used by both the CLI runner and the kit system, and will
+// also be consumed by future orchestrators (remote API, programmatic usage).
+// ---------------------------------------------------------------------------
+/** Type guard: returns true when the step uses a kit (`uses` field present). */
+export function isKitStep(step) {
+    return 'uses' in step && typeof step.uses === 'string';
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@livingdata/pipex",
-  "version": "0.0.3",
+  "version": "0.0.5",
   "description": "Execution engine for containerized pipeline steps",
   "author": "Jérôme Desboeufs <jerome@livingdata.co>",
   "type": "module",
@@ -20,7 +20,8 @@
     "access": "public"
   },
   "scripts": {
-    "start": "tsx src/cli/index.ts",
+    "cli": "tsx src/cli/index.ts",
+    "test": "ava",
     "lint": "xo",
     "lint:fix": "xo --fix",
     "build": "tsc"
@@ -30,11 +31,16 @@
     "commander": "^14.0.3",
     "dotenv": "^17.2.3",
     "execa": "^9.6.1",
+    "lodash-es": "^4.17.23",
     "ora": "^9.3.0",
-    "pino": "^10.3.0"
+    "pino": "^10.3.0",
+    "yaml": "^2.8.2"
   },
   "devDependencies": {
+    "@ava/typescript": "^6.0.0",
+    "@types/lodash-es": "^4.17.12",
     "@types/node": "^25.2.0",
+    "ava": "^6.4.1",
     "tsx": "^4.21.0",
     "xo": "^1.2.3"
   }