builderman 1.5.3 → 1.6.0

package/README.md

@@ -18,6 +18,7 @@ It is designed for monorepos, long-running development processes, and CI/CD pipe
 >   - [Commands & Modes](#commands--modes)
 >   - [Environment Variables](#environment-variables)
 >   - [Dependencies](#dependencies)
+>     - [Command-Level Dependencies](#command-level-dependencies)
 >   - [Pipelines](#pipelines)
 >     - [Concurrency Control](#concurrency-control)
 >     - [Pipeline Composition](#pipeline-composition)
@@ -30,9 +31,15 @@ It is designed for monorepos, long-running development processes, and CI/CD pipe
 > - [Skipping Tasks](#skipping-tasks)
 >   - [Strict Mode](#strict-mode)
 >   - [Task-Level Skip Override](#task-level-skip-override)
+> - [Caching](#caching)
+>   - [Cache Inputs](#cache-inputs)
+>     - [File Paths](#file-paths)
+>     - [Artifacts](#artifacts)
+>     - [Input Resolvers](#input-resolvers)
 > - [Execution Statistics](#execution-statistics)
 >   - [Pipeline Statistics](#pipeline-statistics)
 >   - [Task Statistics](#task-statistics)
+> - [Advanced Examples](#advanced-examples)
 > - [When Should I Use builderman?](#when-should-i-use-builderman)
 
 ## Key Features
@@ -45,6 +52,9 @@ It is designed for monorepos, long-running development processes, and CI/CD pipe
 - 📊 **Rich execution statistics** — always available, even on failure
 - ❌ **Never throws** — failures are returned as structured results
 - 🧱 **Composable pipelines** — pipelines can be converted into tasks
+- 💾 **Task-level caching** — skip tasks when inputs and outputs haven't changed
+- 🎯 **Artifact dependencies** — reference outputs from other tasks in cache inputs
+- 🔌 **Input resolvers** — track package dependencies and other dynamic inputs
 
 ---
 
@@ -63,27 +73,35 @@ import { task, pipeline } from "builderman"
 
 const build = task({
   name: "build",
-  commands: { build: "tsc" },
-  cwd: "packages/my-package", // Optional: defaults to "."
+  commands: {
+    build: "tsc",
+    dev: "tsc --watch",
+  },
 })
 
 const test = task({
   name: "test",
-  commands: { build: "npm test" },
+  commands: {
+    build: "npm test",
+  },
   dependencies: [build],
-  cwd: "packages/my-package",
 })
 
-const result = await pipeline([build, test]).run({
-  command: "build",
+const deploy = task({
+  name: "deploy",
+  commands: {
+    build: "npm run deploy",
+  },
+  dependencies: [test],
 })
 
-if (!result.ok) {
-  console.error("Pipeline failed:", result.error.message)
-}
+const result = await pipeline([build, test, deploy]).run({
+  command: "build",
+})
+console.log(result)
 ```
 
-This defines a simple dependency graph where `test` runs only after `build` completes successfully.
+This defines a simple CI pipeline where `test` runs only after `build` completes, and `deploy` runs only after `test` completes. The result is a structured object with detailed execution statistics.
 
 ---
 
@@ -102,8 +120,8 @@ A **task** represents a unit of work. Each task:
 ```ts
 import { task } from "builderman"
 
-const libTask = task({
-  name: "lib:build",
+const myPackage = task({
+  name: "myPackage",
   commands: {
     build: "tsc",
     dev: {
@@ -111,7 +129,7 @@ const libTask = task({
       readyWhen: (stdout) => stdout.includes("Watching for file changes."),
     },
   },
-  cwd: "packages/lib",
+  cwd: "packages/myPackage",
 })
 ```
 
@@ -133,9 +151,11 @@ Commands may be:
 - A string (executed directly), or
 - An object with:
   - `run`: the command to execute
-  - `readyWhen`: a predicate that marks the task as ready
+  - `dependencies`: optional array of tasks that this command depends on (see [Command-Level Dependencies](#command-level-dependencies))
+  - `readyWhen`: a predicate that marks the task as ready - useful for long-running processes that can allow dependents to start before they exit (e.g. a "watch" process)
   - `teardown`: cleanup logic to run after completion
   - `env`: environment variables specific to this command
+  - `cache`: configuration for task-level caching (see [Caching](#caching))
 
 ---
 
@@ -153,11 +173,11 @@ Environment variables can be provided at multiple levels, with more specific lev
 #### Command-Level Environment Variables
 
 ```ts
-const apiTask = task({
-  name: "api",
+const server = task({
+  name: "server",
   commands: {
     dev: {
-      run: "npm run dev",
+      run: "node server.js",
       env: {
         PORT: "3000",
         NODE_ENV: "development",
@@ -170,15 +190,27 @@ const apiTask = task({
 #### Task-Level Environment Variables
 
 ```ts
-const apiTask = task({
-  name: "api",
-  commands: {
-    dev: "npm run dev",
-    build: "npm run build",
-  },
+const server = task({
+  name: "server",
   env: {
-    API_URL: "http://localhost:3000",
-    LOG_LEVEL: "debug",
+    // in both dev and build, the PORT environment variable will be set to "3000"
+    PORT: "3000",
+  },
+  commands: {
+    dev: {
+      run: "node server.js",
+      env: {
+        LOG_LEVEL: "debug",
+        // overrides the task-level PORT environment variable
+        PORT: "4200",
+      },
+    },
+    build: {
+      run: "node server.js",
+      env: {
+        LOG_LEVEL: "info",
+      },
+    },
   },
 })
 ```
@@ -186,7 +218,7 @@ const apiTask = task({
 #### Pipeline-Level Environment Variables
 
 ```ts
-const result = await pipeline([apiTask]).run({
+const result = await pipeline([server]).run({
   env: {
     DATABASE_URL: "postgres://localhost/mydb",
     REDIS_URL: "redis://localhost:6379",
@@ -225,15 +257,57 @@ In this example, tasks in `innerPipeline` will receive both `INNER_VAR` and `OUT
 
 Tasks may depend on other tasks. A task will not start until all its dependencies have completed (or been skipped).
 
+When a task has task-level dependencies, each command in the task automatically depends on the command with the same name in the dependency task (if it exists). For example, if a task has commands `{ dev, build }` and depends on another task with commands `{ dev, build }`, then this task's `dev` command will depend on the dependency's `dev` command, and this task's `build` command will depend on the dependency's `build` command.
+
 ```ts
-const consumerTask = task({
-  name: "consumer:dev",
+const server = task({
+  name: "server",
   commands: {
-    build: "npm run build",
-    dev: "npm run dev",
+    dev: "node server.js",
+    build: "node server.js",
+  },
+  dependencies: [shared], // Both "build" and "dev" commands will depend on shared's matching commands, if they exist
+})
+```
+
+#### Command-Level Dependencies
+
+You can also specify dependencies at the command level for more granular control. This is useful when different commands have different dependency requirements.
+
+```ts
+const database = task({
+  name: "database",
+  commands: {
+    dev: {
+      run: "docker compose up",
+      readyWhen: (output) => output.includes("ready"),
+      teardown: "docker compose down",
+    },
+  },
+})
+
+const migrations = task({
+  name: "migrations",
+  commands: {
+    build: "npm run migrate",
+  },
+})
+
+const api = task({
+  name: "api",
+  commands: {
+    // Build only needs migrations
+    build: {
+      run: "npm run build",
+      dependencies: [migrations],
+    },
+
+    // Dev needs both the database and migrations
+    dev: {
+      run: "npm run dev",
+      dependencies: [database, migrations],
+    },
   },
-  cwd: "packages/consumer",
-  dependencies: [libTask],
 })
 ```
 
@@ -246,7 +320,7 @@ A **pipeline** executes a set of tasks according to their dependency graph.
 ```ts
 import { pipeline } from "builderman"
 
-const result = await pipeline([libTask, consumerTask]).run({
+const result = await pipeline([backend, frontend]).run({
   command: "dev",
   onTaskBegin: (name) => {
     console.log(`[${name}] starting`)
@@ -289,29 +363,39 @@ If `maxConcurrency` is not specified, there is no limit (tasks run concurrently
 Pipelines can be converted into tasks and composed like any other unit of work.
 
 ```ts
-const build = pipeline([
-  /* ... */
-])
-const test = pipeline([
-  /* ... */
-])
-const deploy = pipeline([
-  /* ... */
-])
+const backend = task({
+  name: "backend",
+  cwd: "packages/backend",
+  commands: { build: "npm run build" },
+})
 
-const buildTask = build.toTask({ name: "build" })
-const testTask = test.toTask({
-  name: "test",
-  dependencies: [buildTask],
-  env: { TEST_ENV: "test-value" }, // Optional: env for nested pipeline
+const frontend = task({
+  name: "frontend",
+  cwd: "packages/frontend",
+  commands: { build: "npm run build" },
+})
+
+const productionMonitoring = task({
+  name: "production-monitoring",
+  cwd: "packages/production-monitoring",
+  commands: { build: "npm run build" },
 })
-const deployTask = deploy.toTask({ name: "deploy", dependencies: [testTask] })
 
-const ci = pipeline([buildTask, testTask, deployTask])
-const result = await ci.run()
+// Convert a pipeline into a task
+const app = pipeline([backend, frontend]).toTask({
+  name: "app",
+  dependencies: [productionMonitoring], // The app task depends on productionMonitoring
+})
+
+const result = await pipeline([app, productionMonitoring]).run()
 ```
 
-When a pipeline is converted to a task, it becomes a **single node** in the dependency graph. The nested pipeline must fully complete before dependents can start.
+When a pipeline is converted to a task:
+
+- It becomes a **single node** in the dependency graph, with the tasks in the pipeline as subtasks
+- The tasks in the pipeline all must either complete or be flagged as 'ready' or 'skipped' before dependents can start
+- You can specify dependencies and environment variables for the pipeline task
+- The tasks in the pipeline are tracked as subtasks in execution statistics, and are included in the summary object
 
 ---
 
@@ -324,7 +408,7 @@ All failures — including task errors, invalid configuration, cancellation, and
 ```ts
 import { pipeline, PipelineError } from "builderman"
 
-const result = await pipeline([libTask, consumerTask]).run()
+const result = await pipeline([backend, frontend]).run()
 
 if (!result.ok) {
   switch (result.error.code) {
@@ -361,7 +445,7 @@ You can cancel a running pipeline using an `AbortSignal`.
 ```ts
 const controller = new AbortController()
 
-const runPromise = pipeline([libTask, consumerTask]).run({
+const runPromise = pipeline([backend, frontend]).run({
   signal: controller.signal,
 })
 
@@ -440,7 +524,10 @@ Teardowns do **not** run when:
 
 ## Skipping Tasks
 
-If a task does not define a command for the current mode, it is **skipped** by default.
+Tasks can be skipped in two scenarios:
+
+1. **Missing command**: If a task does not define a command for the current mode, it is **skipped** by default
+2. **Cache hit**: If a task has cache configuration and the cache matches, the task is **skipped** (see [Caching](#caching))
 
 Skipped tasks:
 
@@ -468,8 +555,12 @@ const apiTask = task({
 
 const result = await pipeline([dbTask, apiTask]).run({
   command: "build",
-  onTaskSkipped: (taskName, mode) => {
-    console.log(`[${taskName}] skipped (no "${mode}" command)`)
+  onTaskSkipped: (taskName, taskId, mode, reason) => {
+    if (reason === "command-not-found") {
+      console.log(`[${taskName}] skipped (no "${mode}" command)`)
+    } else if (reason === "cache-hit") {
+      console.log(`[${taskName}] skipped (cache hit)`)
+    }
   },
 })
 ```
@@ -514,6 +605,193 @@ const result = await pipeline([dbTask]).run({
 
 ---
 
+## Caching
+
+**builderman** supports task-level caching to skip expensive work when inputs and outputs haven't changed. This is useful for build-style tasks where you want to avoid re-running work when nothing has changed.
+
+### Basic Usage
+
+Enable caching by providing `cache` configuration in your command:
+
+```ts
+const buildTask = task({
+  name: "build",
+  commands: {
+    build: {
+      run: "tsc",
+      cache: {
+        inputs: ["src"],
+        // outputs is optional; if omitted, only inputs are tracked
+        outputs: ["dist"],
+      },
+    },
+  },
+})
+```
+
+When caching is enabled:
+
+1. **First run**: The task executes normally and creates a snapshot of the input and output files
+2. **Subsequent runs**: The task compares the current state with the cached snapshot
+3. **Cache hit**: If inputs and outputs are unchanged, the task is **skipped** (no command execution)
+4. **Cache miss**: If anything changed, the task runs and updates the cache
+
+### Cache Inputs
+
+Cache inputs can include:
+
+- **File paths** (strings): Directories or files to track
+- **Artifacts**: References to outputs from other tasks using `task.artifact("command")`
+- **Input resolvers**: Special functions that resolve to cacheable inputs (e.g., package dependencies)
+
+#### File Paths
+
+```ts
+const buildTask = task({
+  name: "build",
+  commands: {
+    build: {
+      run: "tsc",
+      cache: {
+        inputs: ["src", "package.json"],
+        outputs: ["dist"],
+      },
+    },
+  },
+})
+```
+
+#### Artifacts
+
+You can reference outputs from other tasks as cache inputs using `task.artifact("command")`. This creates an artifact dependency that tracks changes to the producing task's outputs.
+
+```ts
+const shared = task({
+  name: "shared",
+  cwd: "packages/shared",
+  commands: {
+    build: {
+      run: "npm run build",
+      cache: {
+        inputs: ["src"],
+        outputs: ["dist"],
+      },
+    },
+  },
+})
+
+const backend = task({
+  name: "backend",
+  cwd: "packages/backend",
+  commands: {
+    build: {
+      run: "npm run build",
+      cache: {
+        inputs: [
+          "src",
+          shared.artifact("build"), // Track changes to shared's build outputs
+        ],
+        outputs: ["dist"],
+      },
+    },
+  },
+})
+```
+
+When using artifacts:
+
+- The artifact-producing task must have `cache.outputs` defined
+- The artifact is included in the cache key, so changes to the artifact invalidate the cache
+- The consuming task automatically depends on the producing task (execution dependency)
+
+#### Input Resolvers
+
+Input resolvers are functions that resolve to cacheable inputs. They're useful for tracking package dependencies and other dynamic inputs.
+
+For example, the `@builderman/resolvers-pnpm` package provides a resolver for pnpm package dependencies:
+
+```ts
+import { task } from "builderman"
+import { pnpm } from "@builderman/resolvers-pnpm"
+
+const server = task({
+  name: "server",
+  cwd: "packages/server",
+  commands: {
+    build: {
+      run: "pnpm build",
+      cache: {
+        inputs: [
+          "src",
+          pnpm.package(), // Automatically tracks pnpm dependencies
+        ],
+        outputs: ["dist"],
+      },
+    },
+  },
+})
+```
+
+The resolver automatically detects whether you're in a workspace or local package and tracks the appropriate `pnpm-lock.yaml` and package dependencies.
+
+### How It Works
+
+The cache system:
+
+- Creates a snapshot of file metadata (modification time and size) for all files in the configured input and output paths
+- For artifacts, tracks the artifact identifier from the producing task's cache
+- For resolvers, includes the resolved input in the cache key
+- Stores snapshots in `.builderman/cache/<version>/` relative to the main process's working directory
+- Compares snapshots before running the task
+- Writes the snapshot **after** successful task completion (ensuring outputs are captured)
+
+### Path Resolution
+
+- Paths may be **absolute** or **relative to the task's `cwd`**
+- Directories are recursively scanned for all files
+- Non-existent paths are treated as empty (no files)
+
+### Cache Information in Statistics
+
+When a task has cache configuration, its statistics include cache information:
+
+```ts
+const result = await pipeline([buildTask]).run()
+
+const taskStats = result.stats.tasks[0]
+
+if (taskStats.cache) {
+  console.log("Cache checked:", taskStats.cache.checked)
+  console.log("Cache hit:", taskStats.cache.hit)
+  console.log("Cache file:", taskStats.cache.cacheFile)
+  console.log("Inputs:", taskStats.cache.inputs)
+  console.log("Outputs:", taskStats.cache.outputs)
+}
+```
+
+### Cache Behavior
+
+- **Cache failures never break execution** — if cache checking fails, the task runs normally
+- **Cache is written after completion** — ensures outputs are captured correctly
+- **Cache is per task and command** — each task-command combination has its own cache file
+- **Cache directory is versioned** — stored under `v1/` to allow future cache format changes
+
+### When to Use Caching
+
+Caching is ideal for:
+
+- Build tasks (TypeScript compilation, bundling, etc.)
+- Code generation tasks
+- Any expensive operation where inputs/outputs can be reliably tracked
+
+Caching is **not** suitable for:
+
+- Tasks that have side effects beyond file outputs
+- Tasks that depend on external state (APIs, databases, etc.)
+- Tasks where outputs are non-deterministic
+
+---
+
 ## Execution Statistics
 
 Every pipeline run returns detailed execution statistics.
@@ -551,6 +829,14 @@ for (const task of result.stats.tasks) {
     console.log("Teardown:", task.teardown.status)
   }
 
+  // Cache information is available when the task has cache configuration
+  if (task.cache) {
+    console.log("Cache checked:", task.cache.checked)
+    if (task.cache.hit !== undefined) {
+      console.log("Cache hit:", task.cache.hit)
+    }
+  }
+
   // when using pipeline.toTask() to convert a pipeline into a task, the task will have subtasks
   if (task.subtasks) {
     for (const subtask of task.subtasks) {
@@ -562,16 +848,201 @@ for (const task of result.stats.tasks) {
 
 ---
 
+## Advanced Examples
+
+### Monorepo Build Pipeline
+
+Here's a comprehensive example showing how to build a complex monorepo pipeline with caching, artifacts, and pipeline composition:
+
+```ts
+import { task, pipeline } from "builderman"
+import { pnpm } from "@builderman/resolvers-pnpm"
+
+/**
+ * Shared core module used by multiple packages
+ */
+const core = task({
+  name: "core",
+  cwd: "packages/core",
+  commands: {
+    build: {
+      run: "pnpm build",
+      cache: {
+        inputs: ["src", pnpm.package()],
+        outputs: ["dist"],
+      },
+    },
+    dev: {
+      run: "pnpm dev",
+      readyWhen: (output) => output.includes("Watching for file changes"),
+    },
+    test: {
+      run: "pnpm test",
+      env: {
+        NODE_ENV: "development",
+      },
+    },
+  },
+})
+
+/**
+ * Factory for related feature packages
+ */
+const createFeatureTask = (name: string) =>
+  task({
+    name,
+    cwd: `packages/${name}`,
+    commands: {
+      build: {
+        run: "pnpm build",
+        cache: {
+          inputs: ["src", core.artifact("build"), pnpm.package()],
+          outputs: ["dist"],
+        },
+      },
+      dev: {
+        run: "pnpm dev",
+        readyWhen: (output) => output.includes("Build complete"),
+      },
+    },
+  })
+
+const featureA = createFeatureTask("feature-a")
+const featureB = createFeatureTask("feature-b")
+
+/**
+ * Compose related features into a single pipeline task
+ */
+const features = pipeline([featureA, featureB]).toTask({
+  name: "features",
+  dependencies: [core],
+})
+
+/**
+ * Consumer package with command-level dependencies
+ */
+const integration = task({
+  name: "integration",
+  cwd: "packages/integration",
+  commands: {
+    build: {
+      run: "pnpm build",
+      cache: {
+        inputs: [
+          "src",
+          core.artifact("build"),
+          featureA.artifact("build"),
+          featureB.artifact("build"),
+          pnpm.package(),
+        ],
+        outputs: ["dist"],
+      },
+    },
+    dev: {
+      run: "pnpm dev",
+      dependencies: [core, features],
+    },
+  },
+})
+
+/**
+ * End-to-end test suites
+ */
+const smokeTests = task({
+  name: "e2e:smoke",
+  cwd: "tests/smoke",
+  commands: {
+    build: {
+      run: "pnpm build",
+      cache: {
+        inputs: [
+          "src",
+          core.artifact("build"),
+          integration.artifact("build"),
+          pnpm.package(),
+        ],
+        outputs: ["dist"],
+      },
+    },
+    test: "pnpm test",
+  },
+  dependencies: [core],
+  env: {
+    NODE_ENV: "development",
+  },
+})
+
+const fullTests = task({
+  name: "e2e:full",
+  cwd: "tests/full",
+  commands: {
+    build: {
+      run: "pnpm build",
+      cache: {
+        inputs: [
+          "src",
+          core.artifact("build"),
+          integration.artifact("build"),
+          pnpm.package(),
+        ],
+        outputs: ["dist"],
+      },
+    },
+    test: "pnpm test",
+  },
+  // Conditional dependency based on environment
+  dependencies: (process.env.CI ? [smokeTests] : []).concat(core),
+  env: {
+    NODE_ENV: "development",
+  },
+})
+
+/**
+ * Pipeline execution
+ */
+const command = process.argv[2]
+
+const result = await pipeline([
+  core,
+  features,
+  integration,
+  smokeTests,
+  fullTests,
+]).run({
+  command,
+  onTaskBegin: (name) => console.log(`[start] ${name}`),
+  onTaskSkipped: (name, _, __, reason) =>
+    console.log(`[skip] ${name} (${reason})`),
+  onTaskComplete: (name) => console.log(`[done] ${name}`),
+})
+
+console.log(result)
+```
+
+This example demonstrates:
+
+- **Caching with artifacts**: Tasks reference outputs from other tasks using `task.artifact("command")`
+- **Input resolvers**: Using `pnpm.package()` to track package dependencies
+- **Pipeline composition**: Converting pipelines to tasks with `pipeline.toTask()`
+- **Command-level dependencies**: Different commands can have different dependencies
+- **Conditional dependencies**: Adjusting dependencies based on runtime conditions
+- **Observability**: Using callbacks to track pipeline execution
+
+---
+
 ## When Should I Use builderman?
 
 **builderman** is a good fit when:
 
-- You have dependent tasks that must run in a strict order
-- You run long-lived dev processes that need readiness detection
-- Cleanup matters (databases, containers, servers)
-- You want structured results instead of log-scraping
+- You have interdependent tasks that must run in a well-defined order
+- You run long-lived processes that need readiness detection (not just exit codes)
+- Cleanup and teardown matter (containers, databases, servers, watchers)
+- You want deterministic execution with structured results instead of log-scraping
+- You need observable pipelines that behave the same locally and in CI
+- You want to compose and reuse workflows, not just run scripts
 
 It may be overkill if:
 
-- You only need a few linear npm scripts
-- You do not need dependency graphs or teardown guarantees
+- Your workflow is a handful of linear npm scripts
+- Tasks are fire-and-forget and don’t require cleanup
+- You don’t need dependency graphs, cancellation, or failure propagation