@openfn/cli 0.0.19 → 0.0.21

package/README.md

@@ -1,69 +1,101 @@
-# @openfn/cli (devtools)
+# @openfn/cli
 
-This package contains a new devtools CLI.
+This package contains a new devtools CLI for running openfn jobs.
 
-The CLI allows you to
+The new CLI includes:
 
-- Run a job (expression), writing output to disk or stdout
-- Compile a job
-- Install modules for jobs
-- Use local language adaptors to run the job
+* A new runtime for executing openfn jobs
+* A new compiler for making openfn jobs runnable
+* Improved, customisable logging output
+* Auto installation of language adaptors
+* Support for the adaptors monorepo
 
-The CLI reads a path as its main argument. That path should either point to a .js file or a folder.
+## Getting Started
 
-- If the path ends in .js, load as a job file and execute. State and output will be read/written relative to it.
-- If the path is a folder, the CLI will look for a job.js, state.json and write an output.json.
+To install:
 
-From this input it will infer a working directory, from which state will be read and output will be written.
-
-All inputs and outputs can be configured with arguments.
+```
+npm install -g @openfn/cli
+```
 
-Run `pnpm openfn -h` to print usage help (the best source of truth right now).
+Make sure everything works by running the built-in test job:
 
-## Usage from the global CLI
+```
+openfn test
+```
 
-See Getting Started for more setup steps.
+Check the version:
 
 ```
-$ npm install -g @openfn/cli`
-$ openfn --help
-$ openfn path/to/expression.js`
+openfn -v
 ```
 
-## Language Adaptors
+Get help:
 
-The old engine used to compile knowlede of langauge adaptors into the job file. We no longer do this.
+```
+openfn help
+```
 
-Generally the new runtime prefers to explictly import all dependencies - although the compiler will auto-insert imports from a given adaptor for you.
+## Basic Usage
 
-You need to pass the name of an adaptor for the runtime to use. It will auto-insert an import statement for you.
+You're probably here to run jobs (expressions), which the CLI makes easy:
 
 ```
-$ openfn job.js -a commmon
+openfn path/to/job.js -ia adaptor-name
 ```
 
-You can use a short-hand name, like above, but longform names also work:
-```
-$ openfn job.js -a @openfn/language-commmon
-```
+You MUST specify which adaptor to use. Pass the `-i` flag to auto-install that adaptor (it's safe to do this redundantly).
 
-You can pass an explicit version number too:
+If output.json and state.json are not passed, the CLI will look for them next to the job.js file. You can pass a path to state by adding `-s path/to/state.json`, and output by passing `-o path/to/output.json`. You can use `-S` and `-O` to pass state through stdin and return the output through stdout.
 
-```
-$ openfn job.js -a commmon@1.7.3
-```
+The CLI can auto-install language adaptors to its own privately maintained repo. Run `openfn repo list` to see where the repo is, and what's in it. Set the `OPENFN_REPO_DIR` env var to specify the repo folder. When autoinstalling, the CLI will check to see if a matching version is found in the repo.
 
-The adaptor also needs to be installed in the CLI's module repo. You can do this manually:
+You can specify adaptors with a shorthand (`http`) or use the full package name (`@openfn/language-http`). You can add a specific version like `http@2.0.0`. You can pass a path to a locally installed adaptor like `http=/repo/openfn/adaptors/my-http-build`. Set the OPENFN_ADAPTORS_REPO env var to load adaptors straight out of the monorepo (pass the `--no-adaptors-repo` flag to disable this for a single run).
 
-```
-$ openfn install commmon
-```
-If no version is provided, the latest will be installed. Again, long and short-form names can be used.
+You can pass `--log info` to get more feedback about what's happening, or `--log debug` for more details than you could ever use.
 
-Alternatively, pass the -i flag when running a job (it's safe to do this redundantly):
-```
-$ openfn job.js -i -a @openfn/language-commmon
-```
+## Advanced Usage
+
+The CLI has actually has a number of commands (the first argument after openfn)
+
+* execute - run a job
+* compile - compile a job to a .js file
+* doc - show documentation for an adaptor function
+* repo - manage the repo of installed modules
+* docgen - generate JSON documentation for an adaptor based on its typescript
+
+If no command is specified, execute will run.
+
+To get more information about a command, including usage examples, run `openfn <command> help`, ie, `openfn compile help`.
+
+## Compilation
+
+The CLI will attempt to compile your job code into normalized Javascript. It will do a number of things to make your code robust and portable:
+
+* The language adaptor will be imported into the file
+* The adaptor's execute function will be exported form the file
+* All top level operations will be added to an array
+* That array will be made the default export of the file
+
+The result of this is a lightweight, modern JS source file. It can be executed in any runtime environment: just execute each function in the exported array.
+
+The CLI uses openfn's own runtime to execute jobs in a safe environment.
+
+All jobs which work against `@openfn/core` will work in the new CLI and runtime environment (note: although this is a work in progress and we are actively looking for help to test this!).
+
+## New Runtime notes
+
+The new openfunction runtime basically does one thing: load a Javascript Module, find the default export, and execute the functions it holds.
+
+So long as your job has an array of functions as its default export, it will run in the new runtime.
+
+# Contributing
+
+First of all, thanks for helping! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock. heart
+
+To get this started, you'll want to clone this repo.
+
+You also need to install `pnpm`.
 
 ## Usage from this repo
 
@@ -78,7 +110,7 @@ See test/execute.test.ts for more usage examples
 
 ## Installing globally
 
-To install the CLI globally from this repo (ie, to do `openfn job.js` instead of `pnpm openfn job.js`), run:
+To install the CLI globally from the build in repo:
 
 ```
 $ npm install -g .
@@ -94,24 +126,15 @@ You should set the OPENFN_REPO_DIR env var to something sensible.
 
 ```
 # In ~/.bashc or whatever
-export OPENFN_REPO_DIR=~/adaptors/@openfn
+export OPENFN_REPO_DIR=~/repo/openfn/cli-repo
 ```
 
-At the time of writing, teh env var name is about to change. Soon you will be able to pass the repo dir into the command line, but the env var is a much easier way to work.
-
-Monorepo support is coming soon.
+To run adaptors straight from the adaptors monorepo:
 
-## Automatic Imports
+export OPENFN_ADAPTORS_REPO=~/repo/openfn/adaptors
 
-The v2 runtime requires explicit imports to be in the job file, or else the job will fail.
-
-The v2 compiler can automatically insert import statements, but it needs to be told which adaptor to use.
-
-```
-$ openfn job.js --adaptors @openfn/language-http
-$ openfn job.js --adaptors @openfn/language-http=path/to/adaptor
-```
+## Contributing changes
 
-If a path is passed (relative to the working directory), that path will be used to load a local version of the adaptor (both at runtime and for import generation)
+Open a PR at https://github.com/openfn/kit. Include a changeset and a description of your change.
 
-If no path is passed, the currently deployed npm package will be used.
+See the root readme for more details about changests,

package/dist/index.js

@@ -4,6 +4,7 @@
 import path from "node:path";
 import * as url from "url";
 import { fork } from "node:child_process";
+import process2 from "node:process";
 function spawn_default(basePath, opts2) {
   const execArgv = [
     "--no-warnings",
@@ -12,13 +13,18 @@ function spawn_default(basePath, opts2) {
   ];
   const dirname = path.dirname(url.fileURLToPath(import.meta.url));
   const child = fork(`${dirname}/process/runner.js`, [], { execArgv });
-  child.on("message", ({ done }) => {
+  child.on("message", ({ done, init, exitCode }) => {
+    if (init) {
+      child.send({ init: true, basePath, opts: opts2 });
+    }
     if (done) {
       child.kill();
-      process.exit(0);
+      process2.exit(exitCode);
     }
   });
-  child.send({ init: true, basePath, opts: opts2 });
+  child.on("close", (code) => {
+    process2.exitCode = code;
+  });
 }
 
 // src/cli.ts
@@ -81,7 +87,7 @@ var list = {
 // src/execute/command.ts
 var executeCommand = {
   command: "execute [path]",
-  desc: "Run an openfn job",
+  desc: `Run an openfn job. Get more help by running openfn <command> help`,
   aliases: ["$0"],
   handler: (argv) => {
     argv.command = "execute";
@@ -170,12 +176,38 @@ var command_default3 = {
   }).example("test", "run the test script").example("test -S 42", "run the test script with state 42")
 };
 
+// src/docgen/command.ts
+var docgenCommand = {
+  command: "docgen <specifier>",
+  desc: "Generate documentation into the repo. Specifier must include a version number.",
+  handler: (argv) => {
+    argv.command = "docgen";
+  },
+  builder: (yargs2) => {
+    return yargs2.example("docgen @openfn/language-common@1.7.5", "");
+  }
+};
+var command_default4 = docgenCommand;
+
+// src/docs/command.ts
+var command_default5 = {
+  command: "docs [adaptor] [operation]",
+  desc: "Print help for an adaptor function. You can use short-hand for adaptor names (ie, common instead of @openfn/language-common)",
+  handler: (argv) => {
+    argv.command = "docs";
+  },
+  builder: (yargs2) => yargs2.example("docs common fn", "Print help for the common fn operation")
+};
+
 // src/cli.ts
-var cmd = yargs(hideBin(process.argv)).command(command_default).command(command_default2).command(install).command(repo).command(command_default3).option("log", {
+var cmd = yargs(hideBin(process.argv)).command(command_default).command(command_default2).command(install).command(repo).command(command_default3).command(command_default5).command(command_default4).option("log", {
   alias: ["l"],
   description: "Set the default log level to none, default, info or debug",
   array: true
-}).alias("v", "version").help();
+}).example("openfn execute help", "Show documentation for the execute command").example(
+  "openfn docs @openfn/language-common each",
+  "Get more help on the common.each command"
+).alias("v", "version").help();
 
 // src/index.ts
 var opts = cmd.parse();

package/dist/process/runner.js

@@ -80,8 +80,10 @@ function ensureOpts(basePath = ".", opts) {
     noCompile: Boolean(opts.noCompile),
     expand: opts.expand !== false,
     outputStdout: Boolean(opts.outputStdout),
+    operation: opts.operation,
     packages: opts.packages,
     stateStdin: opts.stateStdin,
+    specifier: opts.specifier,
     strictOutput: opts.strictOutput ?? true,
     immutable: opts.immutable || false
   };
@@ -156,11 +158,11 @@ var execute_default = (code, state, opts) => {
 function parseAdaptors(opts) {
   const adaptors = {};
   opts.adaptors.reduce((obj, exp) => {
-    const [module, path2] = exp.split("=");
+    const [module, path3] = exp.split("=");
     const { name, version } = getNameAndVersion(module);
     const info = {};
-    if (path2) {
-      info.path = path2;
+    if (path3) {
+      info.path = path3;
     }
     if (version) {
       info.version = version;
@@ -202,10 +204,10 @@ var stripVersionSpecifier = (specifier) => {
   return specifier;
 };
 var resolveSpecifierPath = async (pattern, repoDir, log) => {
-  const [specifier, path2] = pattern.split("=");
-  if (path2) {
-    log.debug(`Resolved ${specifier} to path: ${path2}`);
-    return path2;
+  const [specifier, path3] = pattern.split("=");
+  if (path3) {
+    log.debug(`Resolved ${specifier} to path: ${path3}`);
+    return path3;
   }
   const repoPath = await getModulePath(specifier, repoDir, log);
   if (repoPath) {
@@ -223,15 +225,15 @@ var loadTransformOptions = async (opts, log) => {
     const [pattern] = opts.adaptors;
     const [specifier] = pattern.split("=");
     log.debug(`Attempting to preload typedefs for ${specifier}`);
-    const path2 = await resolveSpecifierPath(pattern, opts.repoDir, log);
-    if (path2) {
+    const path3 = await resolveSpecifierPath(pattern, opts.repoDir, log);
+    if (path3) {
       try {
-        exports = await preloadAdaptorExports(path2);
+        exports = await preloadAdaptorExports(path3);
         if (exports) {
           log.info(`Loaded typedefs for ${specifier}`);
         }
       } catch (e) {
-        log.error(`Failed to load adaptor typedefs from path ${path2}`);
+        log.error(`Failed to load adaptor typedefs from path ${path3}`);
         log.error(e);
       }
     }
@@ -382,10 +384,17 @@ var executeHandler = async (options, logger) => {
   }
   const state = await load_state_default(options, logger);
   const code = await compile_default(options, logger);
-  const result = await execute_default(code, state, options);
-  await serialize_output_default(options, result, logger);
-  const duration = printDuration(new Date().getTime() - start);
-  logger.success(`Done in ${duration}! \u2728`);
+  try {
+    const result = await execute_default(code, state, options);
+    await serialize_output_default(options, result, logger);
+    const duration = printDuration(new Date().getTime() - start);
+    logger.success(`Done in ${duration}! \u2728`);
+  } catch (error) {
+    logger.error(error);
+    const duration = printDuration(new Date().getTime() - start);
+    logger.error(`Took ${duration}.`);
+    process.exitCode = 1;
+  }
 };
 var handler_default = executeHandler;
 
@@ -424,7 +433,165 @@ var testHandler = async (options, logger) => {
 };
 var handler_default3 = testHandler;
 
+// src/docgen/handler.ts
+import { writeFile as writeFile3 } from "node:fs/promises";
+import { readFileSync, writeFileSync, mkdirSync, rmSync } from "node:fs";
+import path2 from "node:path";
+import { describePackage } from "@openfn/describe-package";
+import { getNameAndVersion as getNameAndVersion2 } from "@openfn/runtime";
+var RETRY_DURATION = 500;
+var RETRY_COUNT = 20;
+var TIMEOUT_MS = 1e3 * 60;
+var actualDocGen = (specifier) => describePackage(specifier, {});
+var ensurePath = (filePath) => mkdirSync(path2.dirname(filePath), { recursive: true });
+var generatePlaceholder = (path3) => {
+  writeFileSync(path3, `{ "loading": true, "timestamp": ${Date.now()}}`);
+};
+var finish = (logger, resultPath) => {
+  logger.success("Done! Docs can be found at:\n");
+  logger.print(`  ${path2.resolve(resultPath)}`);
+};
+var generateDocs = async (specifier, path3, docgen, logger) => {
+  const result = await docgen(specifier);
+  await writeFile3(path3, JSON.stringify(result, null, 2));
+  finish(logger, path3);
+  return path3;
+};
+var waitForDocs = async (docs, path3, logger, retryDuration = RETRY_DURATION) => {
+  try {
+    if (docs.hasOwnProperty("loading")) {
+      logger.info("Docs are being loaded by another process. Waiting.");
+      return new Promise((resolve, reject) => {
+        let count = 0;
+        let i = setInterval(() => {
+          logger.info("Waiting..");
+          if (count > RETRY_COUNT) {
+            clearInterval(i);
+            reject(new Error("Timed out waiting for docs to load"));
+          }
+          const updated = JSON.parse(readFileSync(path3, "utf8"));
+          if (!updated.hasOwnProperty("loading")) {
+            logger.info("Docs found!");
+            clearInterval(i);
+            resolve(path3);
+          }
+          count++;
+        }, retryDuration);
+      });
+    } else {
+      logger.info(`Docs already written to cache at ${path3}`);
+      finish(logger, path3);
+      return path3;
+    }
+  } catch (e) {
+    logger.error("Existing doc JSON corrupt. Aborting");
+    throw e;
+  }
+};
+var docgenHandler = (options, logger, docgen = actualDocGen, retryDuration = RETRY_DURATION) => {
+  const { specifier, repoDir } = options;
+  const { version } = getNameAndVersion2(specifier);
+  if (!version) {
+    logger.error("Error: No version number detected");
+    logger.error("eg, @openfn/language-common@1.7.5");
+    logger.error("Aborting");
+    process.exit(9);
+  }
+  logger.success(`Generating docs for ${specifier}`);
+  const path3 = `${repoDir}/docs/${specifier}.json`;
+  ensurePath(path3);
+  const handleError = () => {
+    logger.info("Removing placeholder");
+    rmSync(path3);
+  };
+  try {
+    const existing = readFileSync(path3, "utf8");
+    const json = JSON.parse(existing);
+    if (json && json.timeout && Date.now() - json.timeout >= TIMEOUT_MS) {
+      logger.info(`Expired placeholder found. Removing.`);
+      rmSync(path3);
+      throw new Error("TIMEOUT");
+    }
+    return waitForDocs(json, path3, logger, retryDuration);
+  } catch (e) {
+    if (e.message !== "TIMEOUT") {
+      logger.info(`Docs JSON not found at ${path3}`);
+    }
+    logger.debug("Generating placeholder");
+    generatePlaceholder(path3);
+    return generateDocs(specifier, path3, docgen, logger).catch((e2) => {
+      logger.error("Error generating documentation");
+      logger.error(e2);
+      handleError();
+    });
+  }
+};
+var handler_default4 = docgenHandler;
+
+// src/docs/handler.ts
+import { readFile } from "node:fs/promises";
+import { getNameAndVersion as getNameAndVersion3, getLatestVersion } from "@openfn/runtime";
+var describe = (adaptorName, fn) => `## ${fn.name}(${fn.parameters.map(({ name }) => name).join(",")})
+
+${fn.description}
+
+### Usage Examples
+
+${fn.examples.length ? fn.examples.map((eg) => eg).join("\n\n") : "None"}
+
+### API Reference
+
+https://docs.openfn.org/adaptors/packages/${adaptorName.replace(
+  "@openfn/language-",
+  ""
+)}-docs#${fn.name}
+`;
+var docsHandler = async (options, logger) => {
+  const { adaptor, operation, repoDir } = options;
+  const [adaptorName] = expand_adaptors_default([adaptor], logger);
+  let { name, version } = getNameAndVersion3(adaptorName);
+  if (!version) {
+    logger.info("No version number provided, looking for latest...");
+    version = await getLatestVersion(version);
+    logger.info("Found ", version);
+    logger.success(`Showing docs for ${adaptorName} v${version}`);
+  }
+  logger.info("Generating/loading documentation...");
+  const path3 = await handler_default4(
+    {
+      specifier: `${name}@${version}`,
+      repoDir
+    },
+    createNullLogger()
+  );
+  if (path3) {
+    const source = await readFile(path3, "utf8");
+    const data = JSON.parse(source);
+    const fn = data.functions.find(({ name: name2 }) => name2 === operation);
+    logger.debug("Operation schema:", fn);
+    logger.success(`Documentation for ${name}.${operation} v${version}:
+`);
+    const desc = describe(name, fn);
+    logger.print(desc);
+    logger.success("Done!");
+  } else {
+    logger.error("Not found");
+  }
+};
+var handler_default5 = docsHandler;
+
 // src/commands.ts
+var handlers = {
+  execute: handler_default,
+  compile: handler_default2,
+  test: handler_default3,
+  docgen: handler_default4,
+  docs: handler_default5,
+  ["repo-clean"]: clean,
+  ["repo-install"]: install,
+  ["repo-pwd"]: pwd,
+  ["repo-list"]: list
+};
 var parse = async (basePath, options, log) => {
   const opts = ensureOpts(basePath, options);
   const logger = log || logger_default(CLI, opts);
@@ -439,31 +606,13 @@ var parse = async (basePath, options, log) => {
       "You should set OPENFN_REPO_DIR or pass --repoDir=some/path in to the CLI"
     );
   }
-  let handler = () => null;
-  switch (options.command) {
-    case "repo-install":
-      handler = install;
-      break;
-    case "repo-clean":
-      handler = clean;
-      break;
-    case "repo-pwd":
-      handler = pwd;
-      break;
-    case "repo-list":
-      handler = list;
-      break;
-    case "compile":
-      assertPath(basePath);
-      handler = handler_default2;
-      break;
-    case "test":
-      handler = handler_default3;
-      break;
-    case "execute":
-    default:
-      assertPath(basePath);
-      handler = handler_default;
+  const handler = options.command ? handlers[options.command] : handler_default;
+  if (!opts.command || /^(compile|execute)$/.test(opts.command)) {
+    assertPath(basePath);
+  }
+  if (!handler) {
+    logger.error(`Unrecognise command: ${options.command}`);
+    process.exit(1);
   }
   return handler(opts, logger);
 };
@@ -483,7 +632,10 @@ var assertPath = (basePath) => {
 process.on("message", ({ init, basePath, opts }) => {
   if (init) {
     commands_default(basePath, opts).then(() => {
-      process.send({ done: true });
+      process.send({ done: true, exitCode: process.exitCode });
     });
   }
 });
+process.send({
+  init: true
+});

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@openfn/cli",
-  "version": "0.0.19",
+  "version": "0.0.21",
   "description": "CLI devtools for the openfn toolchain.",
   "engines": {
     "node": ">=16",
@@ -30,15 +30,16 @@
     "@types/yargs": "^17.0.12",
     "ava": "5.1.0",
     "mock-fs": "^5.1.4",
-    "ts-node": "^10.8.1",
+    "ts-node": "^10.9.1",
     "tslib": "^2.4.0",
     "tsup": "^6.2.3",
     "typescript": "^4.7.4"
   },
   "dependencies": {
-    "@openfn/compiler": "^0.0.18",
-    "@openfn/logger": "^0.0.7",
-    "@openfn/runtime": "^0.0.12",
+    "@openfn/compiler": "^0.0.19",
+    "@openfn/describe-package": "^0.0.11",
+    "@openfn/logger": "^0.0.8",
+    "@openfn/runtime": "^0.0.13",
     "fast-safe-stringify": "^2.1.1",
     "rimraf": "^3.0.2",
     "treeify": "^1.1.0",