@db-lyon/flowkit 0.11.1 → 0.12.0

package/docs/configuration.md

@@ -0,0 +1,347 @@
+# Configuration
+
+Flowkit uses YAML files for declarative configuration, with support for layered merging, environment overlays, and schema validation via Zod.
+
+## YAML schema
+
+A flowkit config file has two top-level keys:
+
+```yaml
+tasks:
+  # ...
+flows:
+  # ...
+```
+
+Both default to `{}` if omitted.
+
+### Task definition
+
+```yaml
+tasks:
+  my_task:
+    class_path: path.to.MyTask    # required — how to resolve the task class
+    description: What this task does  # optional
+    group: etl                     # optional — logical grouping label
+    options:                       # optional — default options passed to the task
+      key: value
+```
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `class_path` | `string` | yes | Dotted path to the task class, or a registered name |
+| `description` | `string` | no | Human-readable description |
+| `group` | `string` | no | Logical grouping label |
+| `options` | `object` | no | Default options (merged with step-level overrides) |
+
+### Flow definition
+
+```yaml
+flows:
+  my_flow:
+    description: What this flow does  # required
+    steps:
+      1:
+        task: my_task               # reference a task by name
+        options:                    # optional — override/extend task defaults
+          key: override_value
+      2:
+        flow: other_flow            # reference another flow (nesting)
+      3:
+        task: None                  # skip sentinel — step is always skipped
+```
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `description` | `string` | yes | Human-readable flow description |
+| `steps` | `object` | yes | Steps keyed by number (execution order) |
+
+### Flow step
+
+Each step must have exactly one of `task` or `flow` (mutually exclusive), unless `task: None` is used to mark a skipped step.
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `task` | `string` | one of task/flow | Task name to execute |
+| `flow` | `string` | one of task/flow | Nested flow name to execute |
+| `options` | `object` | no | Override options for this step |
+
+Step numbers are sorted numerically at execution time, so `1, 2, 10` runs in that order (not lexicographic `1, 10, 2`).
+
+### Options merging
+
+When a step executes, options are merged as: **task defaults** + **step overrides** (step wins):
+
+```yaml
+tasks:
+  deploy:
+    class_path: tasks.Deploy
+    options:
+      environment: staging
+      notify: true
+
+flows:
+  release:
+    description: Deploy to production
+    steps:
+      1:
+        task: deploy
+        options:
+          environment: production    # overrides "staging"
+          # notify: true is inherited from task defaults
+```
+
+Runtime parameters passed to `FlowRunner.run({ params })` merge on top with the highest priority (**task defaults < step overrides < runtime params**).
+
+### Step references
+
+Option values may reference the output of earlier steps in the same flow using `${steps.<id>.<path>}`:
+
+```yaml
+flows:
+  chain:
+    description: Pass one step's output into the next
+    steps:
+      1:
+        task: build
+        options:
+          target: plugin
+      2:
+        task: deploy
+        options:
+          artifact: ${steps.1.path}            # whole-value → raw type preserved
+          message:  "deployed ${steps.build.version}"  # embedded → stringified
+```
+
+- **`<id>`** is a step number (`1`) or a task name (`build`, `level.place_actor`). Task names with dots are matched longest-prefix-first.
+- **`<path>`** is a dot path into the step's `result.data`.
+- When a task name appears in multiple steps, references resolve to the **most recently completed** one.
+- A reference that fills the entire string (`"${steps.1.path}"`) is replaced with the raw value, so objects and arrays round-trip. References embedded inside a larger string are stringified.
+- References that can't be resolved throw and fail the step.
+
+References resolve just before the step runs, against the results of already-completed steps in the current flow. Nested flows have their own reference scope — they don't see their parent flow's steps.
+
+### Flow-level hooks
+
+A flow can attach steps that run around the main step sequence, keyed by flow outcome:
+
+```yaml
+flows:
+  deploy:
+    description: Deploy to prod
+    on_start:   [ { task: notify, options: { msg: "starting" } } ]
+    on_success: [ { task: notify, options: { msg: "done ${steps.build.version}" } } ]
+    on_failure: [ { task: notify, options: { msg: "failed: ${error.message}" } } ]
+    finally:    [ { task: cleanup } ]
+    steps:
+      1: { task: build }
+      2: { task: push }
+```
+
+- **`on_start`** runs before any step. Its failure aborts the flow before steps execute.
+- **`on_success`** runs when all steps succeed.
+- **`on_failure`** runs when any step fails. It can reference the error via the `${error.*}` namespace.
+- **`finally`** runs after either outcome, after `on_success`/`on_failure`.
+
+Hook steps share the full step execution model — same task dispatch, same option merging, same runtime params, same `${steps.X.y}` resolution. Inside `on_failure` and `finally`, the `${error.message}`, `${error.name}`, `${error.stack}`, and `${error.step}` references resolve to the failure that triggered them.
+
+Hook failures are captured in `FlowRunResult.hookErrors` but **do not** change the flow's primary success/failure outcome — a failed notifier doesn't rewrite history.
+
+### Per-step retry
+
+A step can retry itself on failure:
+
+```yaml
+steps:
+  1:
+    task: flaky_network_call
+    retries: 3            # up to 4 total attempts
+    retryDelay: 500       # ms between attempts
+    retryOn: "timeout"    # only retry when the error message contains this substring
+```
+
+Omit `retryOn` to retry on any error. The number of attempts taken appears on `FlowStepResult.attempts`.
+
+### Rollback on failure
+
+Mutating tasks may return a `rollback` record on their `TaskResult` pointing to an inverse task:
+
+```ts
+return {
+  success: true,
+  data: { label: 'MyPillar' },
+  rollback: { taskName: 'delete_actor', payload: { label: 'MyPillar' } },
+};
+```
+
+When a flow sets `rollback_on_failure: true` (or the caller passes it on `FlowRunRunOptions`) and a later step fails, the runner invokes the collected rollback records in **reverse order**, best-effort: it continues past individual failures and reports all errors in `FlowRunResult.rollback`.
+
+```yaml
+flows:
+  safe_deploy:
+    description: Deploy with rollback on failure
+    rollback_on_failure: true
+    steps:
+      1: { task: create_thing, options: { label: A } }
+      2: { task: create_thing, options: { label: B } }
+      3: { task: finalize }  # if this fails, thing:B then thing:A are rolled back
+```
+
+Rollback runs after `on_failure` and before `finally`. Nested flow steps' rollback records bubble up to the parent flow so a single `rollback_on_failure` setting covers the whole tree.
+
+The inverse task's configured `options` are resolved for `${ns.path}` references as usual, then the `payload` is merged over them. A payload is runtime data the task recorded, not configuration, so it is passed through **literally** — a `${...}` captured inside one reaches the inverse task unchanged.
+
+Rollback runs outside any step's scope, so those configured `options` resolve against the host namespaces only. A `${steps.…}` or `${error.…}` in an inverse task's defaults has nothing to resolve against and fails that one rollback record (it is reported in `rollback.errors`; the remaining records still run). Keep inverse-task defaults to host namespaces and put step-derived values in the `payload`.
+
+### `agent_prompt` — LLM step
+
+When a `LLMProvider` is attached to the context under `ctx.llm`, the built-in `agent_prompt` task invokes it:
+
+```yaml
+steps:
+  1:
+    task: agent_prompt
+    options:
+      system: "You are a deployment triage agent."
+      prompt: "Last error: ${error.message}. Suggest a fix."
+      model: claude-opus-4-6
+      maxTokens: 512
+      schema: { type: object, properties: { fix: { type: string } } }  # optional
+```
+
+Returns `{ text, parsed?, usage? }`. Provider failures become step failures; missing provider is a clear error.
+
+## Config layering
+
+`loadConfig()` merges up to four layers, left to right:
+
+```
+defaults (code)  →  base file  →  env overlay  →  local overlay
+```
+
+| Layer | Source | Purpose |
+|-------|--------|---------|
+| 1. Defaults | `options.defaults` in code | Hardcoded fallbacks |
+| 2. Base file | `pipeline.yml` | Project-level config (committed) |
+| 3. Env overlay | `pipeline.staging.yml` | Environment-specific overrides |
+| 4. Local overlay | `pipeline.local.yml` | Developer-specific overrides (gitignored) |
+
+### Example
+
+```typescript
+import { loadConfig, EngineConfigSchema } from '@db-lyon/flowkit';
+
+const { config, configDir } = loadConfig({
+  filename: 'pipeline.yml',
+  schema: EngineConfigSchema,
+
+  // Hardcoded defaults merged under everything
+  defaults: {
+    tasks: {},
+    flows: {},
+  },
+
+  // Environment name — loads pipeline.{env}.yml
+  env: process.env.NODE_ENV,
+  // Or read from a specific env var:
+  // envVar: 'APP_ENV',
+
+  // Directory to search (default: cwd)
+  configDir: './config',
+});
+```
+
+The `configDir` return value tells you where the config was loaded from.
+
+### Environment selection
+
+You can specify the environment explicitly or via an env var:
+
+```typescript
+// Explicit
+loadConfig({ filename: 'app.yml', schema, env: 'production' });
+
+// From env var — reads process.env.APP_ENV
+loadConfig({ filename: 'app.yml', schema, envVar: 'APP_ENV' });
+```
+
+If both `env` and `envVar` are provided, `env` takes precedence.
+
+## Deep merge behavior
+
+Config layers are merged using `deepMerge()`, which follows these rules:
+
+| Scenario | Behavior |
+|----------|----------|
+| Objects | Recursive key-by-key merge (override wins per-key) |
+| Arrays | Override replaces the base array |
+| Scalars | Override wins |
+| `null` override | Explicitly nullifies the base value |
+| `undefined` override | No-op (base preserved) |
+
+### Array append mode
+
+By default, arrays in an overlay replace the base array entirely. To append instead, add `__merge: append` to the override array:
+
+```yaml
+# base.yml
+plugins:
+  - eslint
+  - prettier
+
+# base.local.yml
+plugins:
+  - __merge: append
+  - my-custom-plugin
+```
+
+Result: `['eslint', 'prettier', 'my-custom-plugin']`
+
+The `__merge` annotation is stripped from the final array.
+
+## Finding config files
+
+`findConfigFile()` walks up parent directories to locate a file:
+
+```typescript
+import { findConfigFile } from '@db-lyon/flowkit';
+
+const path = findConfigFile('pipeline.yml');
+// Searches cwd, then parent, then grandparent, etc.
+```
+
+Throws if the file isn't found in any ancestor directory.
+
+## Loading raw YAML
+
+For cases where you need the raw parsed YAML without schema validation:
+
+```typescript
+import { loadRawYaml } from '@db-lyon/flowkit';
+
+const data = loadRawYaml('/path/to/file.yml');
+```
+
+## Custom schemas
+
+`EngineConfigSchema` is the minimal schema flowkit needs. You can extend it for your own config sections:
+
+```typescript
+import { z } from 'zod';
+import { EngineConfigSchema } from '@db-lyon/flowkit';
+
+const AppConfigSchema = EngineConfigSchema.extend({
+  database: z.object({
+    host: z.string(),
+    port: z.number().default(5432),
+  }),
+  features: z.record(z.boolean()).default({}),
+});
+
+const { config } = loadConfig({
+  filename: 'app.yml',
+  schema: AppConfigSchema,
+});
+
+// config.tasks, config.flows, config.database, config.features
+```

package/docs/custom-tasks.md

@@ -0,0 +1,232 @@
+# Custom tasks
+
+Tasks are the building blocks of flowkit. Each task is a class that extends `BaseTask` and implements an `execute()` method.
+
+## Anatomy of a task
+
+```typescript
+import { BaseTask, type TaskResult } from '@db-lyon/flowkit';
+
+interface MyOptions {
+  url: string;
+  retries?: number;
+}
+
+export default class FetchData extends BaseTask<MyOptions> {
+  get taskName() {
+    return 'fetch_data';
+  }
+
+  protected validate() {
+    if (!this.options.url) {
+      throw new Error('url option is required');
+    }
+  }
+
+  async execute(): Promise<TaskResult> {
+    const { url, retries = 3 } = this.options;
+
+    const response = await fetch(url);
+    if (!response.ok) {
+      return {
+        success: false,
+        error: new Error(`HTTP ${response.status}`),
+      };
+    }
+
+    const data = await response.json();
+    return {
+      success: true,
+      data: { body: data, status: response.status },
+    };
+  }
+}
+```
+
+### Required members
+
+| Member | Description |
+|--------|-------------|
+| `get taskName()` | A human-readable name used in logging |
+| `execute()` | Async method that performs the work and returns a `TaskResult` |
+
+### Optional members
+
+| Member | Description |
+|--------|-------------|
+| `validate()` | Called before `execute()`. Throw to abort with a validation error. |
+
+### Available on `this`
+
+| Property | Description |
+|----------|-------------|
+| `this.options` | The merged options (task defaults + step overrides), typed as `TOptions` |
+| `this.ctx` | The `TaskContext` passed to the flow runner — read-only, see below |
+| `this.logger` | A child logger scoped to this task instance |
+| `this.resolve(name, options?)` | Build another task by configured name or class path, unexecuted |
+| `this.call(name, options?)` | `resolve()` plus `run()`, returning its `TaskResult` |
+
+## Calling other tasks
+
+`this.call(name)` resolves `name` the same way a flow step does. A configured task name is looked up in the `tasks:` config and dispatched through its `class_path`, inheriting its configured `options` as defaults; anything you pass as `options` merges over them and wins. A name with no configured entry resolves as a class path directly, so a bare `vendor.tasks.Thing` still works.
+
+```yaml
+tasks:
+  soql_query:
+    class_path: caseops.tasks.SoqlQuery
+    options:
+      org: ${org.username}
+```
+
+```typescript
+// Runs caseops.tasks.SoqlQuery with { org: 'admin@example.com', query: 'SELECT ...' }
+const result = await this.call('soql_query', { query: 'SELECT Id FROM Case' });
+```
+
+The `${ns.path}` references in those configured defaults are interpolated for you, against the same scope the calling task itself runs under. The `options` you pass are your own runtime data and are **never** interpolated, so a `${...}` you computed reaches the task verbatim rather than being reinterpreted as configuration.
+
+Calling a task requires a registry on the context, which `FlowRunner` supplies. A task constructed by hand without one throws.
+
+## The task lifecycle
+
+When `task.run()` is called (by the flow runner):
+
+1. `validate()` runs — throw here to reject bad options
+2. `execute()` runs — return a `TaskResult`
+3. The result gets a `duration` field added automatically
+4. If `validate()` or `execute()` throws, the error is caught and returned as `{ success: false, error }`
+
+You never call `run()` yourself in normal usage — the flow runner handles it.
+
+## TaskResult
+
+```typescript
+interface TaskResult {
+  success: boolean;
+  data?: Record<string, unknown>;  // arbitrary output data
+  error?: Error;                    // populated on failure
+  duration?: number;                // milliseconds, set by run()
+}
+```
+
+Return `{ success: true }` for success and `{ success: false, error }` for expected failures. Unexpected exceptions are caught automatically.
+
+## TaskContext
+
+The context carries host-supplied state to every task in a flow run — database connections, API clients, configuration:
+
+```typescript
+const runner = new FlowRunner({
+  // ...
+  context: {
+    logger: myLogger,
+    db: databaseConnection,
+    apiKey: process.env.API_KEY,
+  },
+});
+```
+
+Inside a task:
+
+```typescript
+async execute(): Promise<TaskResult> {
+  const db = this.ctx.db as Database;
+  // ...
+}
+```
+
+Treat the context as read-only. Each task is handed its own derived context, so assigning a key inside a task (`this.ctx.cached = x`) does not reach the next step, another task, or a sub-agent. To share mutable state, put a mutable object on the context up front and write into that:
+
+```typescript
+context: { cache: new Map() }   // this.ctx.cache.set(...) is visible everywhere
+```
+
+## Registering tasks
+
+### By name
+
+```typescript
+const registry = new TaskRegistry();
+registry.register('fetch_data', FetchData as any);
+```
+
+The YAML can then reference it directly:
+
+```yaml
+tasks:
+  fetch_data:
+    class_path: fetch_data
+```
+
+### By class path
+
+```typescript
+registry.registerClassPath('my.tasks.FetchData', FetchData as any);
+```
+
+### Bulk registration
+
+```typescript
+registry.registerAll({
+  fetch_data: FetchData as any,
+  transform: TransformData as any,
+  upload: Upload as any,
+});
+```
+
+### Dynamic resolution
+
+If a `class_path` isn't found in the registry, flowkit converts dots to path separators and looks for a file on disk:
+
+| class_path | Files checked |
+|------------|---------------|
+| `tasks.FetchData` | `tasks/FetchData.ts`, `tasks/FetchData.js`, `tasks/FetchData/index.ts`, `tasks/FetchData/index.js` |
+| `lib.etl.Extract` | `lib/etl/Extract.ts`, `lib/etl/Extract.js`, ... |
+
+The module must have either a `default` export or a named export matching the last segment of the path (e.g., `FetchData`). The export must extend `BaseTask`.
+
+## Built-in: ShellTask
+
+`ShellTask` executes shell commands via `execSync`. Register it under any name you like:
+
+```typescript
+import { ShellTask } from '@db-lyon/flowkit';
+
+registry.register('shell', ShellTask as any);
+```
+
+Then use it in YAML:
+
+```yaml
+tasks:
+  lint:
+    class_path: shell
+    description: Run the linter
+    options:
+      command: npm run lint
+
+  build:
+    class_path: shell
+    description: Build the project
+    options:
+      command: npm run build
+      cwd: /path/to/project
+      timeout: 120000
+```
+
+### ShellTask options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `command` | `string` | (required) | The shell command to execute |
+| `cwd` | `string` | `undefined` | Working directory |
+| `timeout` | `number` | `300000` (5 min) | Timeout in milliseconds |
+
+On success, `result.data.output` contains the trimmed stdout. On failure, `result.data` includes `exitCode`, `stderr`, and `stdout`.
+
+## Listing registered tasks
+
+```typescript
+const names = registry.listRegistered();
+// ['fetch_data', 'shell', 'my.tasks.Transform', ...]
+```