@openfn/cli 0.0.35 → 0.0.38

package/README.md

@@ -1,13 +1,13 @@
 # @openfn/cli
 
-This package contains a new devtools CLI for running openfn jobs.
+This package contains a new devtools CLI for running OpenFn jobs.
 
-The new CLI includes:
+The CLI includes:
 
-* A new runtime for executing openfn jobs
-* A new compiler for making openfn jobs runnable
-* Improved, customisable logging output
-* Auto installation of language adaptors
+* A secure runtime for executing OpenFn jobs and workflows
+* A compiler for making OpenFn jobs runnable
+* Configurable logging output
+* Auto-installation of language adaptors
 * Support for the adaptors monorepo
 
 ## Getting Started
@@ -36,6 +36,22 @@ Get help:
 openfn help
 ```
 
+## Updating
+
+You should be able to install a new version straight on top of your current installation:
+
+```
+npm install -g @openfn/cli
+```
+
+If this fails, try uninstalling the current version first:
+
+```
+npm uninstall -g @openfn/cli
+```
+
+And then re-installing.
+
 ## Migrating from devtools
 
 If you're coming to the CLI from the old openfn devtools, here are a couple of key points to be aware of:
@@ -46,31 +62,37 @@ If you're coming to the CLI from the old openfn devtools, here are a couple of k
 
 ## Basic Usage
 
-You're probably here to run jobs (expressions), which the CLI makes easy:
+You're probably here to run jobs (expressions) or workflows, which the CLI makes easy:
+
 
 ```
+openfn path/to/workflow.json
 openfn path/to/job.js -ia adaptor-name
 ```
 
-You MUST specify which adaptor to use. Pass the `-i` flag to auto-install that adaptor (it's safe to do this redundantly).
+If running a single job, you MUST specify which adaptor to use.
+
+Pass the `-i` flag to auto-install any required adaptors (it's safe to do this redundantly, although the run will be a little slower).
 
-When the job is finished, the CLI will write the `data` property of your state to disk. By default the CLI will create an `output.json` next to the job file. You can pass a path to output by passing `-o path/to/output.json` and state by adding `-s path/to/state.json`. You can use `-S` and `-O` to pass state through stdin and return the output through stdout. To write the entire state object (not just `data`), pass `--no-strict-output`.
+When the finished, the CLI will write the resulting state to disk. By default the CLI will create an `output.json` next to the job file. You can pass a path to output by passing `-o path/to/output.json` and state by adding `-s path/to/state.json`. You can use `-S` and `-O` to pass state through stdin and return the output through stdout.
 
-The CLI can auto-install language adaptors to its own privately maintained repo, just include the `-i` flag in the command and your adaptors will be forever fully managed. Run `openfn repo list` to see where the repo is, and what's in it. Set the `OPENFN_REPO_DIR` env var to specify the repo folder. When autoinstalling, the CLI will check to see if a matching version is found in the repo.
+Note that the CLI will only include the `state.data` key in the output. To write the entire state object (not just `data`), pass `--no-strict-output`.
+
+The CLI maintains a repo for auto-installed adaptors. Run `openfn repo list` to see where the repo is, and what's in it. Set the `OPENFN_REPO_DIR` env var to specify the repo folder. When autoinstalling, the CLI will check to see if a matching version is found in the repo. `openfn repo clean` will remove all adaptors from the repo. The repo also includes any documentation and metadata built with the CLI.
 
 You can specify adaptors with a shorthand (`http`) or use the full package name (`@openfn/language-http`). You can add a specific version like `http@2.0.0`. You can pass a path to a locally installed adaptor like `http=/repo/openfn/adaptors/my-http-build`.
 
-If you have the adaptors monorepo set up on your machine, you can also run adaptors straight from source. Pass the `-m <path>` flag to load from the monorepo. You can also set the monorepo location by setting the `OPENFN_ADAPTORS_REPO` env var to a valid path. After that just include `-m` to load from the monorepo. Remember that adaptors will be loaded from the BUILT package in `dist`, so remember to build an adaptor before running!
+If you have the adaptors monorepo set up on your machine, you can also run adaptors straight from the local build. Pass the `-m <path>` flag to load from the monorepo. You can also set the monorepo location by setting the `OPENFN_ADAPTORS_REPO` env var to a valid path. After that just include `-m` to load from the monorepo. Remember that adaptors will be loaded from the BUILT package in `dist`, so remember to build an adaptor before running!
 
 You can pass `--log info` to get more feedback about what's happening, or `--log debug` for more details than you could ever use.
 
 ## Advanced Usage
 
-The CLI has actually has a number of commands (the first argument after openfn)
+The CLI has a number of commands (the first argument after openfn)
 
 * execute - run a job
 * compile - compile a job to a .js file
-* doc - show documentation for an adaptor function
+* docs - show documentation for an adaptor function
 * repo - manage the repo of installed modules
 * docgen - generate JSON documentation for an adaptor based on its typescript
 
@@ -109,9 +131,42 @@ For a more structured output, you can emit logs as JSON objects with `level`, `n
 ```
 { level: 'info', name: 'CLI', message: ['Loaded adaptor'] }
 ```
-
 Pass `--log-json` to the CLI to do this. You can also set the OPENFN_LOG_JSON env var (and use `--no-log-json` to disable).
 
+## Workflows
+
+As of v0.0.35 the CLI supports running workflows as well as jobs.
+
+A workflow is in execution plan for running several jobs in a sequence. It is defined as a JSON structure.
+
+To see an example workflow, run the test command with `openfn test`.
+
+A workflow has a structure like this (better documentation is coming soon):
+
+```
+{
+  "start": "a", // optionally specify the start node (defaults to jobs[0])
+  "jobs": [
+    {
+      "id": "a",
+      "expression": "fn((state) => state)", // code or a path
+      "adaptor": "@openfn/language-common@1.75", // specifiy the adaptor to use (version optional)
+      "data": {}, // optionally pre-populate the data object (this will be overriden by keys in in previous state)
+      "configuration": {}, // Use this to pass credentials
+      "next": {
+        // This object defines which jobs to call next
+        // All edges returning true will run
+        // If there are no next edges, the workflow will end
+        "b": true,
+        "c": {
+          "condition": "!state.error" // Not that this is an expression, not a function
+        }
+      }
+    },
+  ]
+}
+```
+
 ## Compilation
 
 The CLI will attempt to compile your job code into normalized Javascript. It will do a number of things to make your code robust and portable:
@@ -129,12 +184,6 @@ All jobs which work against `@openfn/core` will work in the new CLI and runtime
 
 If you want to see how the compiler is changing your job, run `openfn compile path/to/job -a <adaptor>` to return the compiled code to stdout. Add `-o path/to/output.js` to save the result to disk.
 
-## New Runtime notes
-
-The new OpenFn runtime will create a secure sandboxed environemtn which loads a Javascript Module, finds the default export, and execute the functions held within it.
-
-So long as your job has an array of functions as its default export, it will run in the new runtime.
-
 # Contributing
 
 First of all, thanks for helping! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock. heart
@@ -170,8 +219,8 @@ The CLI will save and load adaptors from an arbitrary folder on your system.
 
 You should set the OPENFN_REPO_DIR env var to something sensible.
 
+In `~/.bashrc` (or whatever you use), add:
 ```
-# In ~/.bashc or whatever
 export OPENFN_REPO_DIR=~/repo/openfn/cli-repo
 ```
 

package/dist/{chunk-XYZNU5CH.js → chunk-DOKL2XM5.js}

@@ -1,5 +1,5 @@
 // src/util/expand-adaptors.ts
-var expand_adaptors_default = (names) => names?.map((name) => {
+var expand = (name) => {
   if (typeof name === "string") {
     const [left] = name.split("=");
     if (left.match("/") || left.endsWith(".js")) {
@@ -8,7 +8,21 @@ var expand_adaptors_default = (names) => names?.map((name) => {
     return `@openfn/language-${name}`;
   }
   return name;
-});
+};
+var expand_adaptors_default = (opts) => {
+  const { adaptors, workflow } = opts;
+  if (adaptors) {
+    opts.adaptors = adaptors?.map(expand);
+  }
+  if (workflow) {
+    Object.values(workflow.jobs).forEach((job) => {
+      if (job.adaptor) {
+        job.adaptor = expand(job.adaptor);
+      }
+    });
+  }
+  return opts;
+};
 
 // src/util/logger.ts
 import actualCreateLogger, { printDuration } from "@openfn/logger";
@@ -24,13 +38,15 @@ var namespaces = {
   [JOB]: "JOB"
 };
 var createLogger = (name = "", options) => {
-  const logOptions = options.log;
+  const logOptions = options.log || {};
+  let json = false;
   let level = logOptions[name] || logOptions.default || "default";
   if (options.logJson) {
-    logOptions.json = true;
+    json = true;
   }
   return actualCreateLogger(namespaces[name] || name, {
     level,
+    json,
     ...logOptions
   });
 };

package/dist/index.js

@@ -2,7 +2,7 @@
 import {
   DEFAULT_REPO_DIR,
   expand_adaptors_default
-} from "./chunk-XYZNU5CH.js";
+} from "./chunk-DOKL2XM5.js";
 
 // src/process/spawn.ts
 import path from "node:path";
@@ -128,7 +128,7 @@ var adaptors = {
       opts2.adaptors = [];
     }
     if (opts2.expandAdaptors) {
-      opts2.adaptors = expand_adaptors_default(opts2.adaptors);
+      expand_adaptors_default(opts2);
     }
     delete opts2.adaptor;
     delete opts2.a;
@@ -193,19 +193,21 @@ var ignoreImports = {
 };
 var getBaseDir = (opts2) => {
   const basePath = opts2.path ?? ".";
-  if (basePath.endsWith(".js")) {
+  if (/\.(jso?n?)$/.test(basePath)) {
     return path2.dirname(basePath);
   }
   return basePath;
 };
-var jobPath = {
-  name: "job-path",
+var inputPath = {
+  name: "input-path",
   yargs: {
     hidden: true
   },
   ensure: (opts2) => {
     const { path: basePath } = opts2;
-    if (basePath?.endsWith(".js")) {
+    if (basePath?.endsWith(".json")) {
+      opts2.workflowPath = basePath;
+    } else if (basePath?.endsWith(".js")) {
       opts2.jobPath = basePath;
     } else {
       const base = getBaseDir(opts2);
@@ -262,6 +264,13 @@ var repoDir = {
     default: process.env.OPENFN_REPO_DIR || DEFAULT_REPO_DIR
   }
 };
+var start = {
+  name: "start",
+  yargs: {
+    string: true,
+    description: "Specifiy the start node in a workflow"
+  }
+};
 var strictOutput = {
   name: "no-strict-output",
   yargs: {
@@ -325,12 +334,13 @@ var options = [
   compile,
   immutable,
   ignoreImports,
-  jobPath,
+  inputPath,
   logJson,
   outputPath,
   outputStdout,
   repoDir,
   skipAdaptorValidation,
+  start,
   statePath,
   stateStdin,
   strictOutput,
@@ -339,9 +349,9 @@ var options = [
 ];
 var executeCommand = {
   command: "execute [path]",
-  desc: `Run an openfn job. Get more help by running openfn <command> help.
+  desc: `Run an openfn job or workflow. Get more help by running openfn <command> help.
   
-Execute will run a job/expression and write the output state to disk (to ./state.json unless otherwise specified)
+Execute will run a job/workflow at the path and write the output state to disk (to ./state.json unless otherwise specified)
   
 By default only state.data will be written to the output. Include --no-strict-output to write the entire state object.
   
@@ -349,14 +359,14 @@ Remember to include the adaptor name with -a. Auto install adaptors with the -i
   aliases: ["$0"],
   handler: ensure("execute", options),
   builder: (yargs2) => build(options, yargs2).positional("path", {
-    describe: "The path to load the job from (a .js file or a dir containing a job.js file)",
+    describe: "The path to load the job or workflow from (a .js or .json file or a dir containing a job.js file)",
     demandOption: true
   }).example(
     "openfn foo/job.js",
     "Execute foo/job.js with no adaptor and write the final state to foo/job.json"
   ).example(
-    "openfn job.js -ia common",
-    "Execute job.js using @openfn/language-commom , with autoinstall enabled)"
+    "openfn workflow.json -ia common",
+    "Execute workflow.json using @openfn/language-commom (with autoinstall enabled)"
   ).example(
     "openfn job.js -a common --log info",
     "Execute job.js with common adaptor and info-level logging"
@@ -372,7 +382,7 @@ var options2 = [
   expandAdaptors,
   adaptors,
   ignoreImports,
-  jobPath,
+  inputPath,
   logJson,
   override(outputStdout, {
     default: true
@@ -383,32 +393,28 @@ var options2 = [
 ];
 var compileCommand = {
   command: "compile [path]",
-  desc: "Compile an openfn job and print or save the resulting JavaScript.",
+  desc: "Compile an openfn job or workflow and print or save the resulting JavaScript.",
   handler: ensure("compile", options2),
   builder: (yargs2) => build(options2, yargs2).positional("path", {
-    describe: "The path to load the job from (a .js file or a dir containing a job.js file)",
+    describe: "The path to load the job or workflow from (a .js or .json file or a dir containing a job.js file)",
     demandOption: true
   }).example(
-    "compile foo/job.js -O",
-    "Compiles foo/job.js and prints the result to stdout"
+    "compile foo/job.js",
+    "Compiles the job at foo/job.js and prints the result to stdout"
   ).example(
-    "compile foo/job.js -o foo/job-compiled.js",
-    "Compiles foo/job.js and saves the result to foo/job-compiled.js"
+    "compile foo/workflow.json -o foo/workflow-compiled.json",
+    "Compiles the workflow at foo/work.json and prints the result to -o foo/workflow-compiled.json"
   )
 };
 var command_default2 = compileCommand;
 
 // src/test/command.ts
+var options3 = [stateStdin];
 var command_default3 = {
   command: "test",
   desc: "Compiles and runs a test job, printing the result to stdout",
-  handler: (argv) => {
-    argv.command = "test";
-  },
-  builder: (yargs2) => yargs2.option("state-stdin", {
-    alias: "S",
-    description: "Read state from stdin (instead of a file)"
-  }).example("test", "run the test script").example("test -S 42", "run the test script with state 42")
+  handler: ensure("test", options3),
+  builder: (yargs2) => build(options3, yargs2).example("test", "Run the test script")
 };
 
 // src/docgen/command.ts
@@ -435,7 +441,7 @@ var command_default5 = {
 };
 
 // src/metadata/command.ts
-var options3 = [
+var options4 = [
   expandAdaptors,
   adaptors,
   force,
@@ -448,8 +454,8 @@ var options3 = [
 var command_default6 = {
   command: "metadata",
   desc: "Generate metadata for an adaptor config",
-  handler: ensure("metadata", options3),
-  builder: (yargs2) => build(options3, yargs2).example(
+  handler: ensure("metadata", options4),
+  builder: (yargs2) => build(options4, yargs2).example(
     "metadata -a salesforce -s tmp/state.json",
     "Generate salesforce metadata from config in state.json"
   )

package/dist/process/runner.js

@@ -10,54 +10,13 @@ import {
   expand_adaptors_default,
   logger_default,
   printDuration
-} from "../chunk-XYZNU5CH.js";
-
-// src/execute/handler.ts
-import { readFile } from "node:fs/promises";
-
-// src/execute/load-state.ts
-import fs from "node:fs/promises";
-var load_state_default = async (opts, log) => {
-  const { stateStdin, statePath } = opts;
-  log.debug("Load state...");
-  if (stateStdin) {
-    try {
-      const json = JSON.parse(stateStdin);
-      log.success("Read state from stdin");
-      log.debug("state:", json);
-      return json;
-    } catch (e) {
-      log.error("Failed to load state from stdin");
-      log.error(stateStdin);
-      log.error(e);
-      process.exit(1);
-    }
-  }
-  if (statePath) {
-    try {
-      const str = await fs.readFile(statePath, "utf8");
-      const json = JSON.parse(str);
-      log.success(`Loaded state from ${statePath}`);
-      log.debug("state:", json);
-      return json;
-    } catch (e) {
-      log.warn(`Error loading state from ${statePath}`);
-      log.warn(e);
-    }
-  }
-  log.info(
-    "No state provided - using default state { data: {}, configuration: {}"
-  );
-  return {
-    data: {},
-    configuration: {}
-  };
-};
+} from "../chunk-DOKL2XM5.js";
 
 // src/execute/execute.ts
 import run, { getNameAndVersion } from "@openfn/runtime";
-var execute_default = (code, state, opts) => {
-  return run(code, state, {
+var execute_default = (input, state, opts) => {
+  return run(input, state, {
+    start: opts.start,
     timeout: opts.timeout,
     immutableState: opts.immutable,
     logger: logger_default(RUNTIME, opts),
@@ -69,96 +28,41 @@ var execute_default = (code, state, opts) => {
   });
 };
 function parseAdaptors(opts) {
-  const adaptors = {};
-  opts.adaptors.reduce((obj, exp) => {
-    const [module, path5] = exp.split("=");
+  const extractInfo = (specifier) => {
+    const [module, path6] = specifier.split("=");
     const { name, version } = getNameAndVersion(module);
-    const info = {};
-    if (path5) {
-      info.path = path5;
+    const info = {
+      name
+    };
+    if (path6) {
+      info.path = path6;
     }
     if (version) {
       info.version = version;
     }
-    obj[name] = info;
-    return obj;
-  }, adaptors);
-  return adaptors;
-}
-
-// src/compile/compile.ts
-import compile, { preloadAdaptorExports } from "@openfn/compiler";
-import { getModulePath } from "@openfn/runtime";
-var compile_default = async (opts, log) => {
-  log.debug("Loading job...");
-  const compilerOptions = await loadTransformOptions(opts, log);
-  const job = compile(opts.jobSource || opts.jobPath, compilerOptions);
-  if (opts.jobPath) {
-    log.success(`Compiled job from ${opts.jobPath}`);
-  } else {
-    log.success("Compiled job");
-  }
-  return job;
-};
-var stripVersionSpecifier = (specifier) => {
-  const idx = specifier.lastIndexOf("@");
-  if (idx > 0) {
-    return specifier.substring(0, idx);
-  }
-  return specifier;
-};
-var resolveSpecifierPath = async (pattern, repoDir, log) => {
-  const [specifier, path5] = pattern.split("=");
-  if (path5) {
-    log.debug(`Resolved ${specifier} to path: ${path5}`);
-    return path5;
-  }
-  const repoPath = await getModulePath(specifier, repoDir, log);
-  if (repoPath) {
-    return repoPath;
-  }
-  return null;
-};
-var loadTransformOptions = async (opts, log) => {
-  const options = {
-    logger: log || logger_default(COMPILER, opts)
+    return info;
   };
-  if (opts.adaptors?.length && opts.ignoreImports != true) {
-    let exports;
-    const [pattern] = opts.adaptors;
-    const [specifier] = pattern.split("=");
-    log.debug(`Attempting to preload types for ${specifier}`);
-    const path5 = await resolveSpecifierPath(pattern, opts.repoDir, log);
-    if (path5) {
-      try {
-        exports = await preloadAdaptorExports(
-          path5,
-          opts.useAdaptorsMonorepo,
-          log
-        );
-      } catch (e) {
-        log.error(`Failed to load adaptor typedefs from path ${path5}`);
-        log.error(e);
-      }
-    }
-    if (!exports || exports.length === 0) {
-      log.debug(`No module exports found for ${pattern}`);
-    }
-    options["add-imports"] = {
-      ignore: opts.ignoreImports,
-      adaptor: {
-        name: stripVersionSpecifier(specifier),
-        exports,
-        exportAll: true
+  const adaptors = {};
+  if (opts.adaptors) {
+    opts.adaptors.reduce((obj, exp) => {
+      const { name, ...maybeVersionAndPath } = extractInfo(exp);
+      obj[name] = { ...maybeVersionAndPath };
+      return obj;
+    }, adaptors);
+  }
+  if (opts.workflow) {
+    Object.values(opts.workflow.jobs).forEach((job) => {
+      if (job.adaptor) {
+        const { name, ...maybeVersionAndPath } = extractInfo(job.adaptor);
+        adaptors[name] = { ...maybeVersionAndPath };
       }
-    };
+    });
   }
-  return options;
-};
+  return adaptors;
+}
 
 // src/execute/serialize-output.ts
 import { writeFile } from "node:fs/promises";
-import stringify from "fast-safe-stringify";
 var serializeOutput = async (options, result, logger) => {
   let output = result;
   if (output && (output.configuration || output.data)) {
@@ -175,7 +79,7 @@ var serializeOutput = async (options, result, logger) => {
   if (output === void 0) {
     output = "";
   } else {
-    output = stringify(output, void 0, 2);
+    output = JSON.stringify(output, void 0, 2);
   }
   if (options.outputStdout) {
     logger.success(`Result: `);
@@ -188,6 +92,24 @@ var serializeOutput = async (options, result, logger) => {
 };
 var serialize_output_default = serializeOutput;
 
+// src/execute/get-autoinstall-targets.ts
+var getAutoinstallTargets = (options) => {
+  if (options.workflow) {
+    const adaptors = {};
+    Object.values(options.workflow.jobs).forEach((job) => {
+      if (job.adaptor) {
+        adaptors[job.adaptor] = true;
+      }
+    });
+    return Object.keys(adaptors);
+  }
+  if (options.adaptors) {
+    return options.adaptors?.filter((a) => !/=/.test(a));
+  }
+  return [];
+};
+var get_autoinstall_targets_default = getAutoinstallTargets;
+
 // src/repo/handler.ts
 import { exec } from "node:child_process";
 import treeify from "treeify";
@@ -199,9 +121,10 @@ var install = async (opts, log = defaultLogger) => {
     log.success("Installing packages...");
     log.debug("repoDir is set to:", repoDir);
     if (adaptor) {
-      packages = expand_adaptors_default(packages);
+      const expanded = expand_adaptors_default({ adaptors: packages });
+      packages = expanded.adaptors;
     }
-    await rtInstall(packages, repoDir, log);
+    await rtInstall(packages ?? [], repoDir, log);
     const duration = log.timer("install");
     log.success(`Installation complete in ${duration}`);
   }
@@ -258,12 +181,149 @@ var list = async (options, logger) => {
   logger.success("Installed packages:\n\n" + treeify.asTree(output));
 };
 
+// src/compile/compile.ts
+import compile, { preloadAdaptorExports } from "@openfn/compiler";
+import { getModulePath } from "@openfn/runtime";
+var compile_default = async (opts, log) => {
+  log.debug("Compiling...");
+  let job;
+  if (opts.workflow) {
+    job = compileWorkflow(opts.workflow, opts, log);
+  } else {
+    const compilerOptions = await loadTransformOptions(opts, log);
+    job = compile(opts.job || opts.jobPath, compilerOptions);
+  }
+  if (opts.jobPath) {
+    log.success(`Compiled job from ${opts.jobPath}`);
+  } else {
+    log.success("Compiled job");
+  }
+  return job;
+};
+var compileWorkflow = async (workflow, opts, log) => {
+  for (const job of workflow.jobs) {
+    const jobOpts = {
+      ...opts
+    };
+    if (job.adaptor) {
+      jobOpts.adaptors = [job.adaptor];
+    }
+    const compilerOptions = await loadTransformOptions(jobOpts, log);
+    if (job.expression) {
+      job.expression = compile(job.expression, compilerOptions);
+    }
+  }
+  return workflow;
+};
+var stripVersionSpecifier = (specifier) => {
+  const idx = specifier.lastIndexOf("@");
+  if (idx > 0) {
+    return specifier.substring(0, idx);
+  }
+  return specifier;
+};
+var resolveSpecifierPath = async (pattern, repoDir, log) => {
+  const [specifier, path6] = pattern.split("=");
+  if (path6) {
+    log.debug(`Resolved ${specifier} to path: ${path6}`);
+    return path6;
+  }
+  const repoPath = await getModulePath(specifier, repoDir, log);
+  if (repoPath) {
+    return repoPath;
+  }
+  return null;
+};
+var loadTransformOptions = async (opts, log) => {
+  const options = {
+    logger: log || logger_default(COMPILER, opts)
+  };
+  if (opts.adaptors?.length && opts.ignoreImports != true) {
+    let exports;
+    const [pattern] = opts.adaptors;
+    const [specifier] = pattern.split("=");
+    log.debug(`Attempting to preload types for ${specifier}`);
+    const path6 = await resolveSpecifierPath(pattern, opts.repoDir, log);
+    if (path6) {
+      try {
+        exports = await preloadAdaptorExports(
+          path6,
+          opts.useAdaptorsMonorepo,
+          log
+        );
+      } catch (e) {
+        log.error(`Failed to load adaptor typedefs from path ${path6}`);
+        log.error(e);
+      }
+    }
+    if (!exports || exports.length === 0) {
+      log.debug(`No module exports found for ${pattern}`);
+    }
+    options["add-imports"] = {
+      ignore: opts.ignoreImports,
+      adaptor: {
+        name: stripVersionSpecifier(specifier),
+        exports,
+        exportAll: true
+      }
+    };
+  }
+  return options;
+};
+
+// src/util/load-state.ts
+import fs from "node:fs/promises";
+var load_state_default = async (opts, log) => {
+  const { stateStdin, statePath } = opts;
+  log.debug("Loading state...");
+  if (stateStdin) {
+    try {
+      const json = JSON.parse(stateStdin);
+      log.success("Read state from stdin");
+      log.debug("state:", json);
+      return json;
+    } catch (e) {
+      log.error("Failed to load state from stdin");
+      log.error(stateStdin);
+      log.error(e);
+      process.exit(1);
+    }
+  }
+  if (statePath) {
+    try {
+      const str = await fs.readFile(statePath, "utf8");
+      const json = JSON.parse(str);
+      log.success(`Loaded state from ${statePath}`);
+      log.debug("state:", json);
+      return json;
+    } catch (e) {
+      log.warn(`Error loading state from ${statePath}`);
+      log.warn(e);
+    }
+  }
+  log.info(
+    "No state provided - using default state { data: {}, configuration: {} }"
+  );
+  return {
+    data: {},
+    configuration: {}
+  };
+};
+
 // src/util/validate-adaptors.ts
 var validateAdaptors = async (options, logger) => {
   if (options.skipAdaptorValidation) {
     return;
   }
-  if (!options.adaptors || options.adaptors.length === 0) {
+  const hasDeclaredAdaptors = options.adaptors && options.adaptors.length > 0;
+  if (options.workflowPath && hasDeclaredAdaptors) {
+    logger.error("ERROR: adaptor and workflow provided");
+    logger.error(
+      "This is probably not what you meant to do. A workflow should declare an adaptor for each job."
+    );
+    throw new Error("adaptor and workflow provided");
+  }
+  if (!options.workflowPath && !hasDeclaredAdaptors) {
     logger.warn("WARNING: No adaptor provided!");
     logger.warn(
       "This job will probably fail. Pass an adaptor with the -a flag, eg:"
@@ -275,22 +335,76 @@ var validateAdaptors = async (options, logger) => {
 };
 var validate_adaptors_default = validateAdaptors;
 
-// src/execute/handler.ts
-var getAutoinstallTargets = (options) => {
-  if (options.adaptors) {
-    return options.adaptors?.filter((a) => !/=/.test(a));
+// src/util/load-input.ts
+import path from "node:path";
+import fs2 from "node:fs/promises";
+import { isPath } from "@openfn/compiler";
+var load_input_default = async (opts, log) => {
+  log.debug("Loading input...");
+  const { job, workflow, jobPath, workflowPath } = opts;
+  if (workflow || workflowPath) {
+    return loadWorkflow(opts, log);
+  }
+  if (job) {
+    return job;
+  }
+  if (jobPath) {
+    log.debug(`Loading job from ${jobPath}`);
+    opts.job = await fs2.readFile(jobPath, "utf8");
+    return opts.job;
+  }
+};
+var fetchFile = (rootDir, filePath) => {
+  const jobPath = filePath.startsWith("~") ? filePath : path.resolve(rootDir, filePath);
+  return fs2.readFile(jobPath, "utf8");
+};
+var loadWorkflow = async (opts, log) => {
+  const { workflowPath, workflow } = opts;
+  log.debug(`Loading workflow from ${workflowPath}`);
+  try {
+    let wf;
+    let rootDir = opts.baseDir;
+    if (workflowPath) {
+      const workflowRaw = await fs2.readFile(workflowPath, "utf8");
+      wf = JSON.parse(workflowRaw);
+      if (!rootDir) {
+        rootDir = path.dirname(workflowPath);
+      }
+    } else {
+      wf = workflow;
+    }
+    for (const job of wf.jobs) {
+      if (typeof job.expression === "string" && isPath(job.expression)) {
+        job.expression = await fetchFile(rootDir, job.expression);
+      }
+      if (typeof job.configuration === "string" && isPath(job.configuration)) {
+        const configString = await fetchFile(rootDir, job.configuration);
+        job.configuration = JSON.parse(configString);
+      }
+    }
+    opts.workflow = wf;
+    log.debug("Workflow loaded!");
+    return opts.workflow;
+  } catch (e) {
+    log.error(`Error loading workflow from ${workflowPath}`);
+    throw e;
   }
-  return [];
 };
+
+// src/execute/handler.ts
 var executeHandler = async (options, logger) => {
   const start = new Date().getTime();
   await validate_adaptors_default(options, logger);
+  let input = await load_input_default(options, logger);
+  if (options.workflow) {
+    expand_adaptors_default(options);
+  }
   const { repoDir, monorepoPath, autoinstall } = options;
   if (autoinstall) {
     if (monorepoPath) {
       logger.warn("Skipping auto-install as monorepo is being used");
     } else {
-      const autoInstallTargets = getAutoinstallTargets(options);
+      const autoInstallTargets = get_autoinstall_targets_default(options);
       if (autoInstallTargets.length) {
         logger.info("Auto-installing language adaptors");
         await install({ packages: autoInstallTargets, repoDir }, logger);
@@ -298,21 +412,17 @@ var executeHandler = async (options, logger) => {
     }
   }
   const state = await load_state_default(options, logger);
-  let code = "";
   if (options.compile) {
-    code = await compile_default(options, logger);
+    input = await compile_default(options, logger);
   } else {
     logger.info("Skipping compilation as noCompile is set");
-    if (options.jobPath) {
-      code = await readFile(options.jobPath, "utf8");
-      logger.success(`Loaded job from ${options.jobPath} (no compilation)`);
-    }
   }
   try {
-    const result = await execute_default(code, state, options);
+    const result = await execute_default(input, state, options);
     await serialize_output_default(options, result, logger);
     const duration = printDuration(new Date().getTime() - start);
     logger.success(`Done in ${duration}! \u2728`);
+    return result;
   } catch (error) {
     logger.error(error);
     const duration = printDuration(new Date().getTime() - start);
@@ -325,33 +435,65 @@ var handler_default = executeHandler;
 // src/compile/handler.ts
 import { writeFile as writeFile2 } from "node:fs/promises";
 var compileHandler = async (options, logger) => {
-  const code = await compile_default(options, logger);
+  await load_input_default(options, logger);
+  let result = await compile_default(options, logger);
+  if (options.workflow) {
+    result = JSON.stringify(result);
+  }
   if (options.outputStdout) {
     logger.success("Compiled code:");
-    logger.success("\n" + code);
+    logger.success("\n" + result);
   } else {
-    await writeFile2(options.outputPath, code);
+    await writeFile2(options.outputPath, result);
     logger.success(`Compiled to ${options.outputPath}`);
   }
 };
 var handler_default2 = compileHandler;
 
 // src/test/handler.ts
-var sillyMessage = "Calculating the answer to life, the universe, and everything...";
 var testHandler = async (options, logger) => {
   logger.log("Running test job...");
   options.compile = true;
-  options.jobSource = `const fn = () => state => { console.log('${sillyMessage}'); return state * 2; } ; fn()`;
-  delete options.jobPath;
+  options.adaptors = [];
+  options.workflow = {
+    start: "start",
+    jobs: [
+      {
+        id: "start",
+        data: { defaultAnswer: 42 },
+        expression: "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()",
+        next: {
+          calculate: "!state.error"
+        }
+      },
+      {
+        id: "calculate",
+        expression: "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()",
+        next: {
+          result: true
+        }
+      },
+      {
+        id: "result",
+        expression: "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()"
+      }
+    ]
+  };
+  logger.break();
+  logger.info("Workflow object:");
+  logger.info(JSON.stringify(options.workflow, null, 2));
+  logger.break();
   if (!options.stateStdin) {
-    logger.debug("No state provided: try -S <number> to provide some state");
-    options.stateStdin = "21";
+    logger.debug(
+      "No state provided: pass an object with state.data.answer to provide custom input"
+    );
+    logger.debug('eg: -S "{ "data": { "answer": 33 }  }"');
   }
   const silentLogger = createNullLogger();
   const state = await load_state_default(options, silentLogger);
   const code = await compile_default(options, logger);
   const result = await execute_default(code, state, options);
-  logger.success(`Result: ${result}`);
+  logger.success(`Result: ${result.data.answer}`);
   return result;
 };
 var handler_default3 = testHandler;
@@ -359,28 +501,28 @@ var handler_default3 = testHandler;
 // src/docgen/handler.ts
 import { writeFile as writeFile3 } from "node:fs/promises";
 import { readFileSync, writeFileSync, mkdirSync, rmSync } from "node:fs";
-import path from "node:path";
+import path2 from "node:path";
 import { describePackage } from "@openfn/describe-package";
 import { getNameAndVersion as getNameAndVersion2 } from "@openfn/runtime";
 var RETRY_DURATION = 500;
 var RETRY_COUNT = 20;
 var TIMEOUT_MS = 1e3 * 60;
 var actualDocGen = (specifier) => describePackage(specifier, {});
-var ensurePath = (filePath) => mkdirSync(path.dirname(filePath), { recursive: true });
-var generatePlaceholder = (path5) => {
-  writeFileSync(path5, `{ "loading": true, "timestamp": ${Date.now()}}`);
+var ensurePath = (filePath) => mkdirSync(path2.dirname(filePath), { recursive: true });
+var generatePlaceholder = (path6) => {
+  writeFileSync(path6, `{ "loading": true, "timestamp": ${Date.now()}}`);
 };
 var finish = (logger, resultPath) => {
   logger.success("Done! Docs can be found at:\n");
-  logger.print(`  ${path.resolve(resultPath)}`);
+  logger.print(`  ${path2.resolve(resultPath)}`);
 };
-var generateDocs = async (specifier, path5, docgen, logger) => {
+var generateDocs = async (specifier, path6, docgen, logger) => {
   const result = await docgen(specifier);
-  await writeFile3(path5, JSON.stringify(result, null, 2));
-  finish(logger, path5);
-  return path5;
+  await writeFile3(path6, JSON.stringify(result, null, 2));
+  finish(logger, path6);
+  return path6;
 };
-var waitForDocs = async (docs, path5, logger, retryDuration = RETRY_DURATION) => {
+var waitForDocs = async (docs, path6, logger, retryDuration = RETRY_DURATION) => {
   try {
     if (docs.hasOwnProperty("loading")) {
       logger.info("Docs are being loaded by another process. Waiting.");
@@ -392,19 +534,19 @@ var waitForDocs = async (docs, path5, logger, retryDuration = RETRY_DURATION) =>
             clearInterval(i);
             reject(new Error("Timed out waiting for docs to load"));
           }
-          const updated = JSON.parse(readFileSync(path5, "utf8"));
+          const updated = JSON.parse(readFileSync(path6, "utf8"));
           if (!updated.hasOwnProperty("loading")) {
             logger.info("Docs found!");
             clearInterval(i);
-            resolve(path5);
+            resolve(path6);
           }
           count++;
         }, retryDuration);
       });
     } else {
-      logger.info(`Docs already written to cache at ${path5}`);
-      finish(logger, path5);
-      return path5;
+      logger.info(`Docs already written to cache at ${path6}`);
+      finish(logger, path6);
+      return path6;
     }
   } catch (e) {
     logger.error("Existing doc JSON corrupt. Aborting");
@@ -421,28 +563,28 @@ var docgenHandler = (options, logger, docgen = actualDocGen, retryDuration = RET
     process.exit(9);
   }
   logger.success(`Generating docs for ${specifier}`);
-  const path5 = `${repoDir}/docs/${specifier}.json`;
-  ensurePath(path5);
+  const path6 = `${repoDir}/docs/${specifier}.json`;
+  ensurePath(path6);
   const handleError = () => {
     logger.info("Removing placeholder");
-    rmSync(path5);
+    rmSync(path6);
   };
   try {
-    const existing = readFileSync(path5, "utf8");
+    const existing = readFileSync(path6, "utf8");
     const json = JSON.parse(existing);
     if (json && json.timeout && Date.now() - json.timeout >= TIMEOUT_MS) {
       logger.info(`Expired placeholder found. Removing.`);
-      rmSync(path5);
+      rmSync(path6);
       throw new Error("TIMEOUT");
     }
-    return waitForDocs(json, path5, logger, retryDuration);
+    return waitForDocs(json, path6, logger, retryDuration);
   } catch (e) {
     if (e.message !== "TIMEOUT") {
-      logger.info(`Docs JSON not found at ${path5}`);
+      logger.info(`Docs JSON not found at ${path6}`);
     }
     logger.debug("Generating placeholder");
-    generatePlaceholder(path5);
-    return generateDocs(specifier, path5, docgen, logger).catch((e2) => {
+    generatePlaceholder(path6);
+    return generateDocs(specifier, path6, docgen, logger).catch((e2) => {
       logger.error("Error generating documentation");
       logger.error(e2);
       handleError();
@@ -452,7 +594,7 @@ var docgenHandler = (options, logger, docgen = actualDocGen, retryDuration = RET
 var handler_default4 = docgenHandler;
 
 // src/docs/handler.ts
-import { readFile as readFile2 } from "node:fs/promises";
+import { readFile } from "node:fs/promises";
 import { getNameAndVersion as getNameAndVersion3, getLatestVersion } from "@openfn/runtime";
 var describeFn = (adaptorName, fn) => `## ${fn.name}(${fn.parameters.map(({ name }) => name).join(",")})
 
@@ -481,7 +623,8 @@ ${data.functions.map((fn) => `  ${fn.name}(${fn.parameters.map((p) => p.name).jo
 `;
 var docsHandler = async (options, logger) => {
   const { adaptor, operation, repoDir } = options;
-  const [adaptorName] = expand_adaptors_default([adaptor], logger);
+  const { adaptors } = expand_adaptors_default({ adaptors: [adaptor] });
+  const [adaptorName] = adaptors;
   let { name, version } = getNameAndVersion3(adaptorName);
   if (!version) {
     logger.info("No version number provided, looking for latest...");
@@ -490,29 +633,38 @@ var docsHandler = async (options, logger) => {
     logger.success(`Showing docs for ${adaptorName} v${version}`);
   }
   logger.info("Generating/loading documentation...");
-  const path5 = await handler_default4(
+  const path6 = await handler_default4(
     {
       specifier: `${name}@${version}`,
       repoDir
     },
     createNullLogger()
   );
-  if (path5) {
-    const source = await readFile2(path5, "utf8");
+  let didError = false;
+  if (path6) {
+    const source = await readFile(path6, "utf8");
     const data = JSON.parse(source);
     let desc;
     if (operation) {
       const fn = data.functions.find(({ name: name2 }) => name2 === operation);
-      logger.debug("Operation schema:", fn);
-      logger.success(`Documentation for ${name}.${operation} v${version}:
+      if (fn) {
+        logger.debug("Operation schema:", fn);
+        logger.success(`Documentation for ${name}.${operation} v${version}:
 `);
-      desc = describeFn(name, fn);
+        desc = describeFn(name, fn);
+      } else {
+        logger.error(`Failed to find ${operation} in ${name}`);
+      }
     } else {
       logger.debug("No operation provided, listing available operations");
       desc = describeLib(name, data);
     }
     logger.print(desc);
-    logger.success("Done!");
+    if (didError) {
+      logger.error("Error");
+    } else {
+      logger.success("Done!");
+    }
   } else {
     logger.error("Not found");
   }
@@ -522,13 +674,29 @@ var handler_default5 = docsHandler;
 // src/metadata/cache.ts
 import { createHash } from "node:crypto";
 import { readFileSync as readFileSync2 } from "node:fs";
-import path2 from "node:path";
+import path3 from "node:path";
 import { writeFile as writeFile4, mkdir } from "node:fs/promises";
 var getPath = (repoDir, key) => `${repoDir}/meta/${key}.json`;
-var generateKey = (config) => createHash("sha256").update(JSON.stringify(config)).digest("hex");
+var sortKeys = (obj) => {
+  const newObj = {};
+  Object.keys(obj).sort().forEach((k) => {
+    const v = obj[k];
+    if (!v || typeof v == "string" || !isNaN(v) || Array.isArray(v)) {
+      newObj[k] = v;
+    } else {
+      newObj[k] = sortKeys(v);
+    }
+  });
+  return newObj;
+};
+var generateKey = (config, adaptor) => {
+  const sorted = sortKeys(config);
+  const key = `${JSON.stringify(sorted)}-${adaptor}}`;
+  return createHash("sha256").update(key).digest("hex");
+};
 var get = (repoPath, key) => {
   try {
-    const data = readFileSync2(getPath(repoPath, key));
+    const data = readFileSync2(getPath(repoPath, key), "utf8");
     const json = JSON.parse(data);
     return json;
   } catch (e) {
@@ -537,10 +705,10 @@ var get = (repoPath, key) => {
 };
 var set = async (repoPath, key, data) => {
   const fullPath = getPath(repoPath, key);
-  await mkdir(path2.dirname(fullPath), { recursive: true });
+  await mkdir(path3.dirname(fullPath), { recursive: true });
   await writeFile4(fullPath, JSON.stringify(data));
 };
-var cache_default = { get, set, generateKey, getPath };
+var cache_default = { get, set, generateKey, getPath, sortKeys };
 
 // src/metadata/handler.ts
 import { getModuleEntryPoint } from "@openfn/runtime";
@@ -580,8 +748,8 @@ var metadataHandler = async (options, logger) => {
   const adaptor = adaptors[0];
   const state = await load_state_default(options, logger);
   logger.success(`Generating metadata`);
+  logger.info("config:", state);
   const config = state.configuration;
-  logger.info("config:", config);
   if (!config || Object.keys(config).length === 0) {
     logger.error("ERROR: Invalid configuration passed");
     process.exit(1);
@@ -590,7 +758,7 @@ var metadataHandler = async (options, logger) => {
     logger.success("Done!");
     logger.print(cache_default.getPath(repoDir, id));
   };
-  const id = cache_default.generateKey(config);
+  const id = cache_default.generateKey(config, adaptor);
   if (!options.force) {
     logger.debug("config hash: ", id);
     const cached = await cache_default.get(repoDir, id);
@@ -621,13 +789,13 @@ var metadataHandler = async (options, logger) => {
 var handler_default6 = metadataHandler;
 
 // src/util/use-adaptors-repo.ts
-import { readFile as readFile3 } from "node:fs/promises";
-import path3 from "node:path";
+import { readFile as readFile2 } from "node:fs/promises";
+import path4 from "node:path";
 import assert from "node:assert";
-import { getNameAndVersion as getNameAndVersion4 } from "@openfn/runtime";
+import { getNameAndVersion as getNameAndVersion5 } from "@openfn/runtime";
 var validateMonoRepo = async (repoPath, log) => {
   try {
-    const raw = await readFile3(`${repoPath}/package.json`, "utf8");
+    const raw = await readFile2(`${repoPath}/package.json`, "utf8");
     const pkg = JSON.parse(raw);
     assert(pkg.name === "adaptors");
   } catch (e) {
@@ -639,14 +807,14 @@ var updatePath = (adaptor, repoPath, log) => {
   if (adaptor.match("=")) {
     return adaptor;
   }
-  const { name, version } = getNameAndVersion4(adaptor);
+  const { name, version } = getNameAndVersion5(adaptor);
   if (version) {
     log.warn(
       `Warning: Ignoring version specifier on ${adaptor} as loading from the adaptors monorepo`
     );
   }
   const shortName = name.replace("@openfn/language-", "");
-  const abspath = path3.resolve(repoPath, "packages", shortName);
+  const abspath = path4.resolve(repoPath, "packages", shortName);
   return `${name}=${abspath}`;
 };
 var useAdaptorsRepo = async (adaptors, repoPath, log) => {
@@ -663,8 +831,8 @@ var use_adaptors_repo_default = useAdaptorsRepo;
 
 // src/util/print-versions.ts
 import { readFileSync as readFileSync3 } from "node:fs";
-import path4 from "node:path";
-import { getNameAndVersion as getNameAndVersion5 } from "@openfn/runtime";
+import path5 from "node:path";
+import { getNameAndVersion as getNameAndVersion6 } from "@openfn/runtime";
 import { mainSymbols } from "figures";
 var NODE = "node.js";
 var CLI2 = "cli";
@@ -673,7 +841,7 @@ var COMPILER2 = "compiler";
 var { triangleRightSmall: t } = mainSymbols;
 var loadVersionFromPath = (adaptorPath) => {
   try {
-    const pkg = JSON.parse(readFileSync3(path4.resolve(adaptorPath, "package.json"), "utf8"));
+    const pkg = JSON.parse(readFileSync3(path5.resolve(adaptorPath, "package.json"), "utf8"));
     return pkg.version;
   } catch (e) {
     return "unknown";
@@ -691,9 +859,9 @@ var printVersions = async (logger, options = {}) => {
     if (adaptor.match("=")) {
       const [namePart, pathPart] = adaptor.split("=");
       adaptorVersion = loadVersionFromPath(pathPart);
-      adaptorName = getNameAndVersion5(namePart).name;
+      adaptorName = getNameAndVersion6(namePart).name;
     } else {
-      const { name, version: version2 } = getNameAndVersion5(adaptor);
+      const { name, version: version2 } = getNameAndVersion6(adaptor);
       adaptorName = name;
       adaptorVersion = version2 || "latest";
     }
@@ -750,7 +918,7 @@ var handlers = {
   ["repo-list"]: list,
   version: async (opts, logger) => print_versions_default(logger, opts)
 };
-var maybeEnsureOpts = (basePath, options) => /^(execute|compile)$/.test(options.command) ? ensureLogOpts(options) : ensureOpts(basePath, options);
+var maybeEnsureOpts = (basePath, options) => /^(execute|compile|test)$/.test(options.command) ? ensureLogOpts(options) : ensureOpts(basePath, options);
 var parse = async (basePath, options, log) => {
   const opts = maybeEnsureOpts(basePath, options);
   const logger = log || logger_default(CLI, opts);
@@ -771,9 +939,9 @@ var parse = async (basePath, options, log) => {
       logger
     );
   } else if (opts.adaptors && opts.expandAdaptors) {
-    opts.adaptors = expand_adaptors_default(opts.adaptors);
+    expand_adaptors_default(opts);
   }
-  if (/^(test|version)$/.test(opts.command) && !opts.repoDir) {
+  if (!/^(test|version)$/.test(opts.command) && !opts.repoDir) {
     logger.warn(
       "WARNING: no repo module dir found! Using the default (/tmp/repo)"
     );

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@openfn/cli",
-  "version": "0.0.35",
+  "version": "0.0.38",
   "description": "CLI devtools for the openfn toolchain.",
   "engines": {
     "node": ">=18",
@@ -36,15 +36,14 @@
     "typescript": "^4.7.4"
   },
   "dependencies": {
-    "fast-safe-stringify": "^2.1.1",
+    "@openfn/compiler": "0.0.31",
+    "@openfn/describe-package": "0.0.16",
+    "@openfn/logger": "0.0.12",
+    "@openfn/runtime": "0.0.23",
     "figures": "^5.0.0",
     "rimraf": "^3.0.2",
     "treeify": "^1.1.0",
-    "yargs": "^17.5.1",
-    "@openfn/compiler": "0.0.29",
-    "@openfn/logger": "0.0.12",
-    "@openfn/runtime": "0.0.21",
-    "@openfn/describe-package": "0.0.16"
+    "yargs": "^17.5.1"
   },
   "files": [
     "dist",