libdragon 10.3.1 → 10.4.2

package/CHANGELOG.md

@@ -1,5 +1,71 @@
 # Change Log
 
+## [10.4.2] - 2022-04-03
+
+### Fixed
+
+- Make sure actions depending on an `init` fail in a non-project directory to
+keep the project state consistent. This fixes #51.
+- `update` action now tries to update the toolchain image as well. Previously
+this was not the case contrary to what someone would expect. Considering it won't
+change the behaviour for non-latest images and the toolchain did not have any
+breaking changes for a long time, this is not considered a breaking change either.
+- `start` action was printing stuff other than the container id. It doesn't
+anymore.
+- Stop unnecessarily printing container id and a few messages related to updates.
+- Fix a potential race condition that might cause unexpected failures.
+- Correct some errors' exit codes.
+### Added
+
+- A new exit code (`4`) to represent unexpected conditions.
+
+### Changed
+
+- Deprecated providing the image flag for `install` action by displaying a
+warning and removing it from documentation, without changing behaviour even
+though it is higly unlikely this feature was ever used. It mainly exists for
+historical reasons and it wil be removed in next major release.
+- Update documentation to warn against changing strategy is a one way operation.
+- Update documentation to reflect `update` action changes.
+- Minor refactors.
+- Update submodule for local environment.
+
+## [10.4.1] - 2022-03-23
+
+### Fixed
+
+- Update the root makefile to utilize `SOURCE_DIR` for example builds. Then we are
+  able to map container files to local files properly with a generic regex in the
+  problem matcher. This fixes #13 and does not change any behaviour.
+- Add missing examples to the vscode run configurations.
+- Install and build libdragon related things in the container when `exec` and
+  `make` causes a new container run. This was previously prevented on `v10.3.1`
+  because it was unnecessarily delaying all exec operations when the container
+  is started. Refactoring things allowed me to realize this can be improved
+  instead of forcing the user to do a manual `install`.
+- Fix a potential issue that may cause the git commands to run in current folder
+  instead of the project root.
+- Attach the error handler once for spawnProcess.
+- Update vulnerable dependencies.
+
+### Added
+
+- `--directory` option to customize vendoring location.
+- `--strategy` option to select a vendoring strategy. Currently supported options
+  are `submodule`, `subtree` and `manual`. The default is `submodule` and `manual`
+  can be used to opt-out of auto vendoring. Useful if the user wants to utilize
+  a different vendoring strategy and opt-out of the auto-managed git flows.
+
+### Changed
+
+- Migrate to a json file for persistent project information.
+- Only save the configuration file on successful exit except for the initial
+  migration.
+- Do not prevent init if there is a file named libdragon in the target folder.
+  This used to cause problems on windows but I cannot reproduce it anymore
+  with `2.33.1.windows.1`. It may be something caused by my old configuration.
+- Minor performance improvements.
+
 ## [10.3.1] - 2022-01-25
 
 ### Fixed
@@ -19,7 +85,7 @@
 
 ### Changed
 
-- Only accept he image flag for init, install, and update actions as documented.
+- Only accept the image flag for init, install, and update actions as documented.
 - Improve documentation for the `init` and `install` actions.
 - Do not attempt an `install` when running `exec`, just start the container. If
   there is a half-baked container, a manual `install` will potentially restore it.

package/README.md

@@ -22,7 +22,7 @@ To update the library and rebuild/install all the artifacts;
 
     libdragon update
 
-See [next section](#overall-usage) for more details.
+`git` is not strictly required to use this tool as docker's git will be used instead. Still, it is highly recommended to have git on your host machine.
 
 ## Overall usage
 
@@ -38,16 +38,6 @@ You can invoke libdragon as follows;
 
 Run `libdragon help [action]` for more details on individual actions.
 
-### Available flags
-
-**`--image <docker-image>`**
-
-Use this flag to provide a custom image to use instead of the default. It should include the toolchain at `/n64_toolchain`. It will be effective for `init`, `install` and `update` actions and will cause a re-initialization of the container if an image different from what was written to project configuration is provided.
-
-**`--verbose`**
-
-Be verbose. This will print all commands dispatched and their outputs as well.
-
 ## Working on this repository
 
 After cloning this repository on a system with node.js (`>= 14`) & docker (`>= 18`), in this repository's root do;
@@ -92,7 +82,7 @@ To update the submodule and re-build everything;
 
 ### Local test bench
 
-This repository also uses [ed64](https://github.com/anacierdem/ed64), so you can just hit F5 on vscode (The `Run Test Bench` launch configuration) to run the test code in `src` folder to develop libdragon itself quicker if you have an everdrive. There is a caveat though: If you want the problem matcher to work properly, you should name this repository folder `libdragon` exactly.
+This repository also uses [ed64](https://github.com/anacierdem/ed64), so you can just hit F5 on vscode (The `Run Test Bench` launch configuration) to run the test code in `src` folder to develop libdragon itself quicker if you have an everdrive.
 
 There are also additional vscode launch configurations to build libdragon examples and tests based on the currently built and installed libdragon in the docker container. Most of these will always rebuild so that they will use the latest if you made and installed an alternative libdragon. The test bench itself and a few examples (that use the new build system) will already rebuild and reinstall libdragon automatically. These will always produce a rom image using the latest libdragon code in the active repository via its make dependencies. You can clean everything with the `clean` task (open the command palette and choose `Run Task -> clean`).
 

package/index.js

@@ -1,13 +1,22 @@
 #!/usr/bin/env node
 
 const chalk = require('chalk');
-const { readProjectInfo, CommandError, globals } = require('./modules/helpers');
-const actions = require('./modules/actions');
-const { printUsage } = require('./modules/usage');
 
-const STATUS_OK = 0;
-const STATUS_ERROR = 1;
-const STATUS_BAD_PARAM = 2;
+const actions = require('./modules/actions');
+const { fn: printUsage } = require('./modules/actions/help');
+const {
+  STATUS_OK,
+  STATUS_BAD_PARAM,
+  STATUS_ERROR,
+  STATUS_VALIDATION_ERROR,
+} = require('./modules/constants');
+const { globals } = require('./modules/globals');
+const {
+  CommandError,
+  ParameterError,
+  ValidationError,
+} = require('./modules/helpers');
+const { readProjectInfo, writeProjectInfo } = require('./modules/project-info');
 
 let options = {},
   currentAction;
@@ -32,6 +41,22 @@ for (let i = 2; i < process.argv.length; i++) {
     continue;
   }
 
+  if (['--directory', '-d'].includes(val)) {
+    options.VENDOR_DIR = process.argv[++i];
+    continue;
+  } else if (val.indexOf('--directory=') === 0) {
+    options.VENDOR_DIR = val.split('=')[1];
+    continue;
+  }
+
+  if (['--strategy', '-s'].includes(val)) {
+    options.VENDOR_STRAT = process.argv[++i];
+    continue;
+  } else if (val.indexOf('--strategy=') === 0) {
+    options.VENDOR_STRAT = val.split('=')[1];
+    continue;
+  }
+
   if (val.indexOf('--') >= 0) {
     console.error(chalk.red(`Invalid flag \`${val}\``));
     printUsage();
@@ -81,6 +106,15 @@ if (
   process.exit(STATUS_BAD_PARAM);
 }
 
+if (
+  options.VENDOR_STRAT &&
+  !['submodule', 'subtree', 'manual'].includes(options.VENDOR_STRAT)
+) {
+  console.error(chalk.red(`Invalid strategy \`${options.VENDOR_STRAT}\``));
+  printUsage();
+  process.exit(STATUS_BAD_PARAM);
+}
+
 readProjectInfo()
   .then((info) =>
     currentAction.fn(
@@ -93,6 +127,17 @@ readProjectInfo()
     )
   )
   .catch((e) => {
+    if (e instanceof ParameterError) {
+      console.error(chalk.red(e.message));
+      printUsage(undefined, [currentAction.name]);
+      process.exit(STATUS_BAD_PARAM);
+    }
+
+    if (e instanceof ValidationError) {
+      console.error(chalk.red(e.message));
+      process.exit(STATUS_VALIDATION_ERROR);
+    }
+
     const userTargetedError = e instanceof CommandError && e.userCommand;
 
     // Show additional information to user if verbose or we did a mistake
@@ -115,6 +160,10 @@ readProjectInfo()
     // We don't have a user targeted error anymore, we did a mistake for sure
     process.exit(STATUS_ERROR);
   })
+  .then(() => {
+    // Everything was done, update the configuration file if not exiting early
+    return writeProjectInfo();
+  })
   .finally(() => {
     process.exit(STATUS_OK);
   });

package/modules/actions/exec.js

@@ -0,0 +1,97 @@
+const path = require('path');
+
+const { CONTAINER_TARGET_PATH } = require('../constants');
+const { log, dockerExec, toPosixPath } = require('../helpers');
+
+const { start } = require('./start');
+const {
+  dockerHostUserParams,
+  installDependencies,
+  mustHaveProject,
+} = require('./utils');
+
+function dockerRelativeWorkdir(libdragonInfo) {
+  return (
+    CONTAINER_TARGET_PATH +
+    '/' +
+    toPosixPath(path.relative(libdragonInfo.root, process.cwd()))
+  );
+}
+
+function dockerRelativeWorkdirParams(libdragonInfo) {
+  return ['--workdir', dockerRelativeWorkdir(libdragonInfo)];
+}
+
+const exec = async (libdragonInfo, commandAndParams) => {
+  await mustHaveProject(libdragonInfo);
+
+  log(
+    `Running ${commandAndParams[0]} at ${dockerRelativeWorkdir(
+      libdragonInfo
+    )} with [${commandAndParams.slice(1)}]`,
+    true
+  );
+
+  const tryCmd = (libdragonInfo) =>
+    libdragonInfo.containerId &&
+    dockerExec(
+      libdragonInfo,
+      [
+        ...dockerRelativeWorkdirParams(libdragonInfo),
+        ...dockerHostUserParams(libdragonInfo),
+      ],
+      commandAndParams,
+      true,
+      true // Cannot use "full" here, we need to know if the container is alive
+    );
+
+  let started = false;
+  const startOnceAndCmd = async () => {
+    if (!started) {
+      const newId = await start(libdragonInfo);
+      started = true;
+
+      // Re-install vendors on new container if one was created upon start
+      // Ideally we would want the consumer to handle dependencies and rebuild
+      // libdragon if necessary. Currently this saves the day with a little bit
+      // extra waiting when the container is deleted.
+      if (libdragonInfo.containerId !== newId) {
+        await installDependencies({
+          ...libdragonInfo,
+          containerId: newId,
+        });
+      }
+      await tryCmd({
+        ...libdragonInfo,
+        containerId: newId,
+      });
+      return newId;
+    }
+  };
+
+  if (!libdragonInfo.containerId) {
+    log(`Container does not exist for sure, restart`, true);
+    await startOnceAndCmd();
+    return;
+  }
+
+  try {
+    await tryCmd(libdragonInfo);
+  } catch (e) {
+    if (
+      !e.out ||
+      // TODO: is there a better way?
+      !e.out.toString().includes(libdragonInfo.containerId)
+    ) {
+      throw e;
+    }
+    await startOnceAndCmd();
+  }
+};
+
+module.exports = {
+  name: 'exec',
+  fn: exec,
+  forwardsRestParams: true,
+  showStatus: true,
+};

package/modules/actions/help.js

@@ -0,0 +1,163 @@
+const chalk = require('chalk');
+const commandLineUsage = require('command-line-usage');
+
+const { log } = require('../helpers');
+
+const printUsage = (_, actionArr) => {
+  const globalOptionDefinitions = [
+    {
+      name: 'verbose',
+      description:
+        'Be verbose. This will print all commands dispatched and their outputs as well. Will also start printing error stack traces.',
+      alias: 'v',
+      typeLabel: ' ',
+      group: 'global',
+    },
+  ];
+
+  const optionDefinitions = [
+    {
+      name: 'image',
+      description:
+        'Use this flag to provide a custom image to use instead of the default. It should include the toolchain at `/n64_toolchain`. It will cause a re-initialization of the container if a different image is provided.\n',
+      alias: 'i',
+      typeLabel: '<docker-image>',
+      group: 'docker',
+    },
+    {
+      name: 'directory',
+      description: `Directory where libdragon files are expected. It must be a project relative path as it will be mounted on the docker container. The cli will create and manage it when using a non-manual strategy. Defaults to \`./libdragon\` if not provided.\n`,
+      alias: 'd',
+      typeLabel: '<path>',
+      group: 'vendoring',
+    },
+    {
+      name: 'strategy',
+      description: `libdragon Vendoring strategy. Defaults to \`submodule\`, which safely creates a git repository at project root and a submodule at \`--directory\` to automatically update the vendored libdragon files.
+
+        With \`subtree\`, the cli will create a subtree at \`--directory\` instead. Keep in mind that git user name and email must be set up for this to work. Do not use if you are not using git yourself.
+
+        To disable auto-vendoring, init with \`manual\`. With \`manual\`, libdragon files are expected at the location provided by \`--directory\` flag and the user is responsible for vendoring and updating them. This will allow using any other manual vendoring method.
+
+        You can always switch to manual by re-running \`init\` with \`--strategy manual\`, though you will be responsible for managing the existing submodule/subtree. Also it is not possible to automatically switch back.
+
+        With the \`manual\` strategy, it is still recommended to have a git repository at project root such that container actions can execute faster by caching the container id inside the \`.git\` folder.\n`,
+      alias: 'v',
+      typeLabel: '<strategy>',
+      group: 'vendoring',
+    },
+  ];
+
+  // TODO: move these to respective actions
+
+  const actions = {
+    help: {
+      name: 'help [action]',
+      summary: 'Display this help information or details for the given action.',
+    },
+    init: {
+      name: 'init',
+      summary: 'Create a libdragon project in the current directory.',
+      description: `Creates a libdragon project in the current directory. Every libdragon project will have its own docker container instance. If you are in a git repository or an NPM project, libdragon will be initialized at their root also marking there with a \`.libdragon\` folder. Do not remove the \`.libdragon\` folder and commit its contents if you are using source control, as it keeps persistent libdragon project information.
+
+      By default, a git repository and a submodule at \`./libdragon\` will be created to automatically update the vendored libdragon files on subsequent \`update\`s. If you intend to opt-out from this feature, see the \`--strategy manual\` flag to provide your self-managed libdragon copy. The default behaviour is intended for users who primarily want to consume libdragon as is.
+
+      If this is the first time you are creating a libdragon project at that location, this action will also create skeleton project files to kickstart things with the given image, if provided. For subsequent runs, it will act like \`start\` thus can be used to revive an existing project without modifying it.`,
+      group: ['docker', 'vendoring'],
+    },
+    make: {
+      name: 'make [params]',
+      summary: 'Run the libdragon build system in the current directory.',
+      description: `Runs the libdragon build system in the current directory. It will mirror your current working directory to the container.
+
+      Must be run in an initialized libdragon project. This action is a shortcut to the \`exec\` action under the hood.`,
+    },
+    exec: {
+      name: 'exec <command>',
+      summary: 'Execute given command in the current directory.',
+      description: `Executes the given command in the container passing down any arguments provided. If you change your host working directory, the command will be executed in the corresponding folder in the container as well.
+
+      This action will first try to execute the command in the container and if the container is not accessible, it will attempt a complete \`start\` cycle.
+
+      This will properly passthrough your TTY if you have one. So by running \`libdragon exec bash\` you can start an interactive bash session with full TTY support.
+      
+      Must be run in an initialized libdragon project.`,
+    },
+    start: {
+      name: 'start',
+      summary: 'Start the container for current project.',
+      description: `Start the container assigned to the current libdragon project. Will first attempt to start an existing container if found, followed by a new container run and installation similar to the \`install\` action. Will always print out the container id on success.
+
+        Must be run in an initialized libdragon project.`,
+    },
+    name: {
+      name: 'stop',
+      summary: 'Stop the container for current project.',
+      description: `Stop the container assigned to the current libdragon project.
+
+        Must be run in an initialized libdragon project.`,
+    },
+    install: {
+      name: 'install',
+      summary: 'Vendor libdragon as is.',
+      description: `Attempts to build and install everything libdragon related into the container. This includes all the tools and third parties used by libdragon except for the toolchain. If you have made changes to libdragon, you can execute this action to build everything based on your changes. Requires you to have an intact vendoring target (also see the \`--directory\` flag). If you are not working on libdragon itself, you can just use the \`update\` action instead.
+
+      Must be run in an initialized libdragon project. This can be useful to recover from a half-baked container.`,
+    },
+    update: {
+      name: 'update',
+      summary: 'Update libdragon and do an install.',
+      description: `Will update the docker image and if you are using auto-vendoring (see \`--strategy\`), will also update the submodule/subtree from the remote branch (\`trunk\`) with a merge/squash strategy and then perform a \`libdragon install\`. You can use the \`install\` action to only update all libdragon related artifacts in the container.
+
+        Must be run in an initialized libdragon project.`,
+      group: ['docker'],
+    },
+  };
+
+  const actionsToShow = actionArr
+    ?.filter((action) => Object.keys(actions).includes(action))
+    .filter((action) => !['help'].includes(action));
+
+  const sections = [
+    {
+      header: chalk.green('Usage:'),
+      content: 'libdragon [flags] <action>',
+    },
+    ...(actionsToShow?.length
+      ? actionsToShow.flatMap((action) => [
+          {
+            header: chalk.green(`${action} action:`),
+            content: actions[action].description,
+          },
+          actions[action].group
+            ? {
+                header: `accepted flags:`,
+                optionList: optionDefinitions,
+                group: actions[action].group,
+              }
+            : {},
+        ])
+      : [
+          {
+            header: chalk.green('Available Commands:'),
+            content: Object.values(actions).map((action) => ({
+              name: action.name,
+              summary: action.summary,
+            })),
+          },
+        ]),
+    {
+      header: chalk.green('Global flags:'),
+      optionList: globalOptionDefinitions,
+    },
+  ];
+  const usage = commandLineUsage(sections);
+  log(usage);
+};
+
+module.exports = {
+  name: 'help',
+  fn: printUsage,
+  showStatus: true,
+  forwardsRestParams: true,
+};

package/modules/actions/index.js

@@ -0,0 +1,25 @@
+const { fn: exec } = require('./exec');
+
+const make = async (libdragonInfo, params) => {
+  await exec(libdragonInfo, ['make', ...params]);
+};
+
+// TODO: separate into files
+module.exports = {
+  start: require('./start'),
+  stop: require('./stop'),
+  init: require('./init'),
+
+  exec: require('./exec'),
+  make: {
+    name: 'make',
+    fn: make,
+    forwardsRestParams: true,
+    showStatus: true,
+  },
+
+  install: require('./install'),
+  update: require('./update'),
+
+  help: require('./help'),
+};