@openfn/cli 0.0.35 → 0.0.39

package/README.md

@@ -1,13 +1,13 @@
 # @openfn/cli
 
-This package contains a new devtools CLI for running openfn jobs.
+This package contains a new devtools CLI for running OpenFn jobs.
 
-The new CLI includes:
+The CLI includes:
 
-* A new runtime for executing openfn jobs
-* A new compiler for making openfn jobs runnable
-* Improved, customisable logging output
-* Auto installation of language adaptors
+* A secure runtime for executing OpenFn jobs and workflows
+* A compiler for making OpenFn jobs runnable
+* Configurable logging output
+* Auto-installation of language adaptors
 * Support for the adaptors monorepo
 
 ## Getting Started
@@ -36,41 +36,60 @@ Get help:
 openfn help
 ```
 
+## Updating
+
+You should be able to install a new version straight on top of your current installation:
+
+```
+npm install -g @openfn/cli
+```
+
+If this fails, try uninstalling the current version first:
+
+```
+npm uninstall -g @openfn/cli
+```
+
+And then re-installing.
+
 ## Migrating from devtools
 
 If you're coming to the CLI from the old openfn devtools, here are a couple of key points to be aware of:
 
 * The CLI has a shorter, sleeker syntax, so your command should be much shorter
 * The CLI will automatically install adaptors for you (with full version control)
-* By default, the CLI will only write state.data to output. This is to encourage better state management. Pass `--no-strict-output` to save the entire state object.
 
 ## Basic Usage
 
-You're probably here to run jobs (expressions), which the CLI makes easy:
+You're probably here to run jobs (expressions) or workflows, which the CLI makes easy:
+
 
 ```
+openfn path/to/workflow.json
 openfn path/to/job.js -ia adaptor-name
 ```
 
-You MUST specify which adaptor to use. Pass the `-i` flag to auto-install that adaptor (it's safe to do this redundantly).
+If running a single job, you MUST specify which adaptor to use.
+
+Pass the `-i` flag to auto-install any required adaptors (it's safe to do this redundantly, although the run will be a little slower).
 
-When the job is finished, the CLI will write the `data` property of your state to disk. By default the CLI will create an `output.json` next to the job file. You can pass a path to output by passing `-o path/to/output.json` and state by adding `-s path/to/state.json`. You can use `-S` and `-O` to pass state through stdin and return the output through stdout. To write the entire state object (not just `data`), pass `--no-strict-output`.
+When the finished, the CLI will write the resulting state to disk. By default the CLI will create an `output.json` next to the job file. You can pass a path to output by passing `-o path/to/output.json` and state by adding `-s path/to/state.json`. You can use `-S` and `-O` to pass state through stdin and return the output through stdout.
 
-The CLI can auto-install language adaptors to its own privately maintained repo, just include the `-i` flag in the command and your adaptors will be forever fully managed. Run `openfn repo list` to see where the repo is, and what's in it. Set the `OPENFN_REPO_DIR` env var to specify the repo folder. When autoinstalling, the CLI will check to see if a matching version is found in the repo.
+The CLI maintains a repo for auto-installed adaptors. Run `openfn repo list` to see where the repo is, and what's in it. Set the `OPENFN_REPO_DIR` env var to specify the repo folder. When autoinstalling, the CLI will check to see if a matching version is found in the repo. `openfn repo clean` will remove all adaptors from the repo. The repo also includes any documentation and metadata built with the CLI.
 
 You can specify adaptors with a shorthand (`http`) or use the full package name (`@openfn/language-http`). You can add a specific version like `http@2.0.0`. You can pass a path to a locally installed adaptor like `http=/repo/openfn/adaptors/my-http-build`.
 
-If you have the adaptors monorepo set up on your machine, you can also run adaptors straight from source. Pass the `-m <path>` flag to load from the monorepo. You can also set the monorepo location by setting the `OPENFN_ADAPTORS_REPO` env var to a valid path. After that just include `-m` to load from the monorepo. Remember that adaptors will be loaded from the BUILT package in `dist`, so remember to build an adaptor before running!
+If you have the adaptors monorepo set up on your machine, you can also run adaptors straight from the local build. Pass the `-m <path>` flag to load from the monorepo. You can also set the monorepo location by setting the `OPENFN_ADAPTORS_REPO` env var to a valid path. After that just include `-m` to load from the monorepo. Remember that adaptors will be loaded from the BUILT package in `dist`, so remember to build an adaptor before running!
 
 You can pass `--log info` to get more feedback about what's happening, or `--log debug` for more details than you could ever use.
 
 ## Advanced Usage
 
-The CLI has actually has a number of commands (the first argument after openfn)
+The CLI has a number of commands (the first argument after openfn)
 
 * execute - run a job
 * compile - compile a job to a .js file
-* doc - show documentation for an adaptor function
+* docs - show documentation for an adaptor function
 * repo - manage the repo of installed modules
 * docgen - generate JSON documentation for an adaptor based on its typescript
 
@@ -109,9 +128,42 @@ For a more structured output, you can emit logs as JSON objects with `level`, `n
 ```
 { level: 'info', name: 'CLI', message: ['Loaded adaptor'] }
 ```
-
 Pass `--log-json` to the CLI to do this. You can also set the OPENFN_LOG_JSON env var (and use `--no-log-json` to disable).
 
+## Workflows
+
+As of v0.0.35 the CLI supports running workflows as well as jobs.
+
+A workflow is in execution plan for running several jobs in a sequence. It is defined as a JSON structure.
+
+To see an example workflow, run the test command with `openfn test`.
+
+A workflow has a structure like this (better documentation is coming soon):
+
+```
+{
+  "start": "a", // optionally specify the start node (defaults to jobs[0])
+  "jobs": [
+    {
+      "id": "a",
+      "expression": "fn((state) => state)", // code or a path
+      "adaptor": "@openfn/language-common@1.75", // specifiy the adaptor to use (version optional)
+      "data": {}, // optionally pre-populate the data object (this will be overriden by keys in in previous state)
+      "configuration": {}, // Use this to pass credentials
+      "next": {
+        // This object defines which jobs to call next
+        // All edges returning true will run
+        // If there are no next edges, the workflow will end
+        "b": true,
+        "c": {
+          "condition": "!state.error" // Not that this is an expression, not a function
+        }
+      }
+    },
+  ]
+}
+```
+
 ## Compilation
 
 The CLI will attempt to compile your job code into normalized Javascript. It will do a number of things to make your code robust and portable:
@@ -129,12 +181,6 @@ All jobs which work against `@openfn/core` will work in the new CLI and runtime
 
 If you want to see how the compiler is changing your job, run `openfn compile path/to/job -a <adaptor>` to return the compiled code to stdout. Add `-o path/to/output.js` to save the result to disk.
 
-## New Runtime notes
-
-The new OpenFn runtime will create a secure sandboxed environemtn which loads a Javascript Module, finds the default export, and execute the functions held within it.
-
-So long as your job has an array of functions as its default export, it will run in the new runtime.
-
 # Contributing
 
 First of all, thanks for helping! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock. heart
@@ -170,8 +216,8 @@ The CLI will save and load adaptors from an arbitrary folder on your system.
 
 You should set the OPENFN_REPO_DIR env var to something sensible.
 
+In `~/.bashrc` (or whatever you use), add:
 ```
-# In ~/.bashc or whatever
 export OPENFN_REPO_DIR=~/repo/openfn/cli-repo
 ```
 

package/dist/{chunk-XYZNU5CH.js → chunk-LV5XDERP.js}

@@ -1,5 +1,5 @@
 // src/util/expand-adaptors.ts
-var expand_adaptors_default = (names) => names?.map((name) => {
+var expand = (name) => {
   if (typeof name === "string") {
     const [left] = name.split("=");
     if (left.match("/") || left.endsWith(".js")) {
@@ -8,7 +8,21 @@ var expand_adaptors_default = (names) => names?.map((name) => {
     return `@openfn/language-${name}`;
   }
   return name;
-});
+};
+var expand_adaptors_default = (opts) => {
+  const { adaptors, workflow } = opts;
+  if (adaptors) {
+    opts.adaptors = adaptors?.map(expand);
+  }
+  if (workflow) {
+    Object.values(workflow.jobs).forEach((job) => {
+      if (job.adaptor) {
+        job.adaptor = expand(job.adaptor);
+      }
+    });
+  }
+  return opts;
+};
 
 // src/util/logger.ts
 import actualCreateLogger, { printDuration } from "@openfn/logger";
@@ -24,13 +38,15 @@ var namespaces = {
   [JOB]: "JOB"
 };
 var createLogger = (name = "", options) => {
-  const logOptions = options.log;
+  const logOptions = options.log || {};
+  let json = false;
   let level = logOptions[name] || logOptions.default || "default";
   if (options.logJson) {
-    logOptions.json = true;
+    json = true;
   }
   return actualCreateLogger(namespaces[name] || name, {
     level,
+    json,
     ...logOptions
   });
 };
@@ -107,7 +123,6 @@ function ensureOpts(basePath = ".", opts) {
     skipAdaptorValidation: opts.skipAdaptorValidation ?? false,
     specifier: opts.specifier,
     stateStdin: opts.stateStdin,
-    strictOutput: opts.strictOutput ?? true,
     timeout: opts.timeout
   };
   const set = (key, value) => {

package/dist/index.js

@@ -2,7 +2,7 @@
 import {
   DEFAULT_REPO_DIR,
   expand_adaptors_default
-} from "./chunk-XYZNU5CH.js";
+} from "./chunk-LV5XDERP.js";
 
 // src/process/spawn.ts
 import path from "node:path";
@@ -128,7 +128,7 @@ var adaptors = {
       opts2.adaptors = [];
     }
     if (opts2.expandAdaptors) {
-      opts2.adaptors = expand_adaptors_default(opts2.adaptors);
+      expand_adaptors_default(opts2);
     }
     delete opts2.adaptor;
     delete opts2.a;
@@ -193,19 +193,21 @@ var ignoreImports = {
 };
 var getBaseDir = (opts2) => {
   const basePath = opts2.path ?? ".";
-  if (basePath.endsWith(".js")) {
+  if (/\.(jso?n?)$/.test(basePath)) {
     return path2.dirname(basePath);
   }
   return basePath;
 };
-var jobPath = {
-  name: "job-path",
+var inputPath = {
+  name: "input-path",
   yargs: {
     hidden: true
   },
   ensure: (opts2) => {
     const { path: basePath } = opts2;
-    if (basePath?.endsWith(".js")) {
+    if (basePath?.endsWith(".json")) {
+      opts2.workflowPath = basePath;
+    } else if (basePath?.endsWith(".js")) {
       opts2.jobPath = basePath;
     } else {
       const base = getBaseDir(opts2);
@@ -262,14 +264,38 @@ var repoDir = {
     default: process.env.OPENFN_REPO_DIR || DEFAULT_REPO_DIR
   }
 };
+var start = {
+  name: "start",
+  yargs: {
+    string: true,
+    description: "Specifiy the start node in a workflow"
+  }
+};
 var strictOutput = {
   name: "no-strict-output",
   yargs: {
+    deprecated: true,
+    hidden: true,
+    boolean: true
+  },
+  ensure: (opts2) => {
+    if (!opts2.hasOwnProperty("strict")) {
+      opts2.strict = opts2.strictOutput;
+    }
+    delete opts2.strictOutput;
+  }
+};
+var strict = {
+  name: "no-strict",
+  yargs: {
+    default: false,
     boolean: true,
-    description: "Allow properties other than data to be returned in the output"
+    description: "Strict state handling, meaning only state.data is returned from a job."
   },
   ensure: (opts2) => {
-    setDefaultValue(opts2, "strictOutput", true);
+    if (!opts2.hasOwnProperty("strictOutput")) {
+      setDefaultValue(opts2, "strict", false);
+    }
   }
 };
 var skipAdaptorValidation = {
@@ -325,38 +351,40 @@ var options = [
   compile,
   immutable,
   ignoreImports,
-  jobPath,
+  inputPath,
   logJson,
   outputPath,
   outputStdout,
   repoDir,
   skipAdaptorValidation,
+  start,
   statePath,
   stateStdin,
+  strict,
   strictOutput,
   timeout,
   useAdaptorsMonorepo
 ];
 var executeCommand = {
   command: "execute [path]",
-  desc: `Run an openfn job. Get more help by running openfn <command> help.
+  desc: `Run an openfn job or workflow. Get more help by running openfn <command> help.
   
-Execute will run a job/expression and write the output state to disk (to ./state.json unless otherwise specified)
+Execute will run a job/workflow at the path and write the output state to disk (to ./state.json unless otherwise specified)
   
-By default only state.data will be written to the output. Include --no-strict-output to write the entire state object.
+By default only state.data will be returned fron a job. Include --no-strict to write the entire state object.
   
 Remember to include the adaptor name with -a. Auto install adaptors with the -i flag.`,
   aliases: ["$0"],
   handler: ensure("execute", options),
   builder: (yargs2) => build(options, yargs2).positional("path", {
-    describe: "The path to load the job from (a .js file or a dir containing a job.js file)",
+    describe: "The path to load the job or workflow from (a .js or .json file or a dir containing a job.js file)",
     demandOption: true
   }).example(
     "openfn foo/job.js",
     "Execute foo/job.js with no adaptor and write the final state to foo/job.json"
   ).example(
-    "openfn job.js -ia common",
-    "Execute job.js using @openfn/language-commom , with autoinstall enabled)"
+    "openfn workflow.json -ia common",
+    "Execute workflow.json using @openfn/language-commom (with autoinstall enabled)"
   ).example(
     "openfn job.js -a common --log info",
     "Execute job.js with common adaptor and info-level logging"
@@ -372,7 +400,7 @@ var options2 = [
   expandAdaptors,
   adaptors,
   ignoreImports,
-  jobPath,
+  inputPath,
   logJson,
   override(outputStdout, {
     default: true
@@ -383,32 +411,28 @@ var options2 = [
 ];
 var compileCommand = {
   command: "compile [path]",
-  desc: "Compile an openfn job and print or save the resulting JavaScript.",
+  desc: "Compile an openfn job or workflow and print or save the resulting JavaScript.",
   handler: ensure("compile", options2),
   builder: (yargs2) => build(options2, yargs2).positional("path", {
-    describe: "The path to load the job from (a .js file or a dir containing a job.js file)",
+    describe: "The path to load the job or workflow from (a .js or .json file or a dir containing a job.js file)",
     demandOption: true
   }).example(
-    "compile foo/job.js -O",
-    "Compiles foo/job.js and prints the result to stdout"
+    "compile foo/job.js",
+    "Compiles the job at foo/job.js and prints the result to stdout"
   ).example(
-    "compile foo/job.js -o foo/job-compiled.js",
-    "Compiles foo/job.js and saves the result to foo/job-compiled.js"
+    "compile foo/workflow.json -o foo/workflow-compiled.json",
+    "Compiles the workflow at foo/work.json and prints the result to -o foo/workflow-compiled.json"
   )
 };
 var command_default2 = compileCommand;
 
 // src/test/command.ts
+var options3 = [stateStdin];
 var command_default3 = {
   command: "test",
   desc: "Compiles and runs a test job, printing the result to stdout",
-  handler: (argv) => {
-    argv.command = "test";
-  },
-  builder: (yargs2) => yargs2.option("state-stdin", {
-    alias: "S",
-    description: "Read state from stdin (instead of a file)"
-  }).example("test", "run the test script").example("test -S 42", "run the test script with state 42")
+  handler: ensure("test", options3),
+  builder: (yargs2) => build(options3, yargs2).example("test", "Run the test script")
 };
 
 // src/docgen/command.ts
@@ -435,7 +459,7 @@ var command_default5 = {
 };
 
 // src/metadata/command.ts
-var options3 = [
+var options4 = [
   expandAdaptors,
   adaptors,
   force,
@@ -448,8 +472,8 @@ var options3 = [
 var command_default6 = {
   command: "metadata",
   desc: "Generate metadata for an adaptor config",
-  handler: ensure("metadata", options3),
-  builder: (yargs2) => build(options3, yargs2).example(
+  handler: ensure("metadata", options4),
+  builder: (yargs2) => build(options4, yargs2).example(
     "metadata -a salesforce -s tmp/state.json",
     "Generate salesforce metadata from config in state.json"
   )