libdragon 10.2.0 → 10.4.0

package/CHANGELOG.md

@@ -1,5 +1,93 @@
 # Change Log
 
+## [10.4.0] - 2022-03-23
+
+### Fixed
+
+- Update the root makefile to utilize `SOURCE_DIR` for example builds. Then we are
+  able to map container files to local files properly with a generic regex in the
+  problem matcher. This fixes #13 and does not change any behaviour.
+- Add missing examples to the vscode run configurations.
+- Install and build libdragon related things in the container when `exec` and
+  `make` causes a new container run. This was previously prevented on `v10.3.1`
+  because it was unnecessarily delaying all exec operations when the container
+  is started. Refactoring things allowed me to realize this can be improved
+  instead of forcing the user to do a manual `install`.
+- Fix a potential issue that may cause the git commands to run in current folder
+  instead of the project root.
+- Attach the error handler once for spawnProcess.
+- Update vulnerable dependencies.
+
+### Added
+
+- `--directory` option to customize vendoring location.
+- `--strategy` option to select a vendoring strategy. Currently supported options
+  are `submodule`, `subtree` and `manual`. The default is `submodule` and `manual`
+  can be used to opt-out of auto vendoring. Useful if the user wants to utilize
+  a different vendoring strategy and opt-out of the auto-managed git flows.
+
+### Changed
+
+- Migrate to a json file for persistent project information.
+- Only save the configuration file on successful exit except for the initial
+  migration.
+- Do not prevent init if there is a file named libdragon in the target folder.
+  This used to cause problems on windows but I cannot reproduce it anymore
+  with `2.33.1.windows.1`. It may be something caused by my old configuration.
+- Minor performance improvements.
+
+## [10.3.1] - 2022-01-25
+
+### Fixed
+
+- Do not try to parse arguments after exec/make. They used to get evaluated as
+  libdragon paramaters previously, preventing passing down -v to the container
+  make for example.
+- Docker image update issues
+  - Attempt an image update whenever the image flag is provided. Previously this
+    was only done if a different image name is provided, preventing the update of
+    the latest image. Previously not providing an image name was behaving the same
+    so this is not a breaking change.
+  - Start action used to update the image if provided but this bug was already
+    prevented by previous fixes by not accepting the image flag for actions other
+    than init/update/install. Start action no longer tries to update the image
+    regardless.
+
+### Changed
+
+- Only accept the image flag for init, install, and update actions as documented.
+- Improve documentation for the `init` and `install` actions.
+- Do not attempt an `install` when running `exec`, just start the container. If
+  there is a half-baked container, a manual `install` will potentially restore it.
+- Update libdragon.
+
+### Added
+
+- Extra information for skipping the image flag when doing init for an already
+  initialized project.
+
+## [10.3.0] - 2022-01-20
+
+### Changed
+
+- Update dependencies.
+- Detailed help output.
+- Move action descriptions to `libdragon help`.
+- Update libdragon to latest version for local build.
+
+### Added
+
+- Shorthand flag support.
+
+## [10.2.1] - 2021-10-14
+
+### Changed
+
+- Updated ed64.
+
+### Fixed
+
+- Fix skeleton project to match latest libdragon.
 
 ## [10.2.0] - 2021-10-10
 
@@ -7,42 +95,42 @@
 
 - Container discovery. The tool can now find a container even if `.git` is lost.
 - A few additional error messages for some potentially confusing cases such as
-already having a file with `libdragon` like name.
+  already having a file with `libdragon` like name.
 
 ### Changed
 
 - Show more output for downloading the container and initial git operations.
 - Remove an extra log during initialization.
 - The submodule was always being initialized when a container is started. This
-was making some actions inconsistent. For example `install` or `make` action
-was trying to re-initialize the submodule unnecessarily. This is fixed by only
-initializing it with the `init` action. If any of those need to re-init the
-container, now they assume there is an intact libdragon folder to use.
+  was making some actions inconsistent. For example `install` or `make` action
+  was trying to re-initialize the submodule unnecessarily. This is fixed by only
+  initializing it with the `init` action. If any of those need to re-init the
+  container, now they assume there is an intact libdragon folder to use.
 - Similarly a git repository is not initialized unnecessarily anymore.
 - `update` and `install` are now able to start containers if necessary.
 - Always try to copy skeleton files, they won't overwrite anything already.
 - Do not re-initialize if there is a `.libdragon` folder. We now only try to
-start it in this case. If it is not a complete container, it can probably be
-recovered by a `libdragon install` or `libdragon update`.
+  start it in this case. If it is not a complete container, it can probably be
+  recovered by a `libdragon install` or `libdragon update`.
 
 ### Fixed
 
 - Fix wording for libdragon install on the container.
 - Improve image name persitence such that the tool finds and updates it more
-consistently.
+  consistently.
 
 ## [10.1.0] - 2021-10-07
 
 ### Added
 
 - `exec` action to execute arbitrary commands in the container with TTY support.
-This also improves the color output support as docker now knows it is using TTY.
+  This also improves the color output support as docker now knows it is using TTY.
 - Add more verbose logging for skeleton project copy.
 
 ### Changed
 
 - Removed partially not working `--byte-swap` option. It does not already work
-with the new libdragon build system so there is no need keeping it in the tool.
+  with the new libdragon build system so there is no need keeping it in the tool.
 
 ### Fixed
 
@@ -53,14 +141,14 @@ with the new libdragon build system so there is no need keeping it in the tool.
 ### Changed
 
 - A complete re-write of the tool. Check documentation for the new usage. It is
-much more straightforward to use now. `libdragon make` behaves almost the same.
+  much more straightforward to use now. `libdragon make` behaves almost the same.
 
 ## [9.0.0] - 2021-09-06
 
 ### Changed
 
 - Updated libdragon. We will be changing the update mechanism to be based on a
-git pull, so listing them here will not be useful anymore, let's not bother now.
+  git pull, so listing them here will not be useful anymore, let's not bother now.
 - Start using the new build system.
 - Update node engine spec to be at least 14.
 - Added minimum docker version to readme.
@@ -110,7 +198,6 @@ git pull, so listing them here will not be useful anymore, let's not bother now.
 - Support for RTC status/read/write commands (https://github.com/DragonMinded/libdragon/pull/152)
 - Generic libdragon NPM script
 
-
 ## [7.0.0] - 2021-06-06
 
 ### Changed

package/README.md

@@ -22,7 +22,7 @@ To update the library and rebuild/install all the artifacts;
 
     libdragon update
 
-See [next section](#overall-usage) for more details.
+`git` is not strictly required to use this tool as docker's git will be used instead. Still, it is highly recommended to have git on your host machine.
 
 ## Overall usage
 
@@ -36,55 +36,7 @@ You can invoke libdragon as follows;
 
     libdragon [flags] <action>
 
-### Available actions
-
-__`init`__
-
-Creates a libdragon project in the current directory. Every libdragon project will have its own docker container instance. If you are in a git repository or an NPM project, libdragon will be initialized at their root also marking there with a `.libdragon` folder.
-
-A git repository and a submodule at `./libdragon` will also be created. Do not remove the `.libdragon` folder and commit its contents if you are using git, as it keeps persistent libdragon project information.
-
-If this is the first time you are creating a libdragon project at that location, this action will also create skeleton project files to kickstart things.
-
-__`make`__
-
-Runs the libdragon build system in the current directory. It will mirror your current working directory to the container.
-
-This action is a shortcut to the `exec` action under the hood.
-
-__`exec`__
-
-Executes the given command in the container passing down any arguments provided. If you change your host working directory, the command will be executed in the corresponding folder in the container as well.
-
-This action will first try to execute the command in the container and if the container is not accessible, it will attempt a complete `start` cycle.
-
-This will properly passthrough your TTY if you have one. So by running `libdragon exec bash` you can start an interactive bash session with full TTY support.
-
-__`install`__
-
-Attempts to build and install everything libdragon related into the container. This includes all the tools and third parties used by libdragon except for the toolchain. If you have made changes to libdragon, you can execute this action to build everything based on your changes. Requires you to have an intact `libdragon` at the root of the project. If you are not working on libdragon, you can just use the `update` action instead.
-
-__`update`__
-
-This action will update the submodule from the remote branch (`trunk`) with a merge strategy and then perform a `libdragon install`. You can use the `install` action to only update all libdragon related artifacts in the container.
-
-__`start`__
-
-Start the container assigned to the current libdragon project. Will first attempt to start an existing container if found, followed by a new container run and installation similar to the `install` action. Will always print out the container id on success.
-
-__`stop`__
-
-Stop the container assigned to the current libdragon project.
-
-### Available flags
-
-__`--image <docker-image>`__
-
-Use this flag to provide a custom image to use instead of the default. It should include the toolchain at `/n64_toolchain`. It will be effective for `init`, `install` and `update` actions and will cause a re-initialization of the container if an image different from what was written to project configuration is provided.
-
-__`--verbose`__
-
-Be verbose. This will print all commands dispatched and their outputs as well.
+Run `libdragon help [action]` for more details on individual actions.
 
 ## Working on this repository
 
@@ -130,7 +82,7 @@ To update the submodule and re-build everything;
 
 ### Local test bench
 
-This repository also uses [ed64](https://github.com/anacierdem/ed64), so you can just hit F5 on vscode (The `Run Test Bench` launch configuration) to run the test code in `src` folder to develop libdragon itself quicker if you have an everdrive. There is a caveat though: If you want the problem matcher to work properly, you should name this repository folder `libdragon` exactly.
+This repository also uses [ed64](https://github.com/anacierdem/ed64), so you can just hit F5 on vscode (The `Run Test Bench` launch configuration) to run the test code in `src` folder to develop libdragon itself quicker if you have an everdrive.
 
 There are also additional vscode launch configurations to build libdragon examples and tests based on the currently built and installed libdragon in the docker container. Most of these will always rebuild so that they will use the latest if you made and installed an alternative libdragon. The test bench itself and a few examples (that use the new build system) will already rebuild and reinstall libdragon automatically. These will always produce a rom image using the latest libdragon code in the active repository via its make dependencies. You can clean everything with the `clean` task (open the command palette and choose `Run Task -> clean`).
 

package/index.js

@@ -1,91 +1,132 @@
 #!/usr/bin/env node
 
 const chalk = require('chalk');
-const { readProjectInfo, CommandError, globals } = require('./modules/helpers');
-const actions = require('./modules/actions');
-
-const STATUS_OK = 0;
-const STATUS_ERROR = 1;
-const STATUS_BAD_PARAM = 2;
-
-// Command line options
-const options = {
-  DOCKER_IMAGE: undefined,
-  VERBOSE: false,
 
-  ACTION: undefined,
-  PARAMS: undefined,
-};
-
-// Allow standard io here
-/* eslint-disable no-console */
+const actions = require('./modules/actions');
+const { fn: printUsage } = require('./modules/actions/help');
+const {
+  STATUS_OK,
+  STATUS_BAD_PARAM,
+  STATUS_ERROR,
+} = require('./modules/constants');
+const { globals } = require('./modules/globals');
+const { CommandError, ParameterError } = require('./modules/helpers');
+const { readProjectInfo, writeProjectInfo } = require('./modules/project-info');
+
+let options = {},
+  currentAction;
 
 for (let i = 2; i < process.argv.length; i++) {
   const val = process.argv[i];
 
-  if (val === '--verbose') {
+  // Allow console here
+  /* eslint-disable no-console */
+
+  if (['--verbose', '-v'].includes(val)) {
     options.VERBOSE = true;
     globals.verbose = true;
     continue;
   }
 
-  if (val === '--image') {
+  if (['--image', '-i'].includes(val)) {
     options.DOCKER_IMAGE = process.argv[++i];
     continue;
+  } else if (val.indexOf('--image=') === 0) {
+    options.DOCKER_IMAGE = val.split('=')[1];
+    continue;
+  }
+
+  if (['--directory', '-d'].includes(val)) {
+    options.VENDOR_DIR = process.argv[++i];
+    continue;
+  } else if (val.indexOf('--directory=') === 0) {
+    options.VENDOR_DIR = val.split('=')[1];
+    continue;
+  }
+
+  if (['--strategy', '-s'].includes(val)) {
+    options.VENDOR_STRAT = process.argv[++i];
+    continue;
+  } else if (val.indexOf('--strategy=') === 0) {
+    options.VENDOR_STRAT = val.split('=')[1];
+    continue;
   }
 
   if (val.indexOf('--') >= 0) {
     console.error(chalk.red(`Invalid flag \`${val}\``));
-    actions.help.fn();
+    printUsage();
     process.exit(STATUS_BAD_PARAM);
   }
 
-  if (options.ACTION) {
+  if (currentAction) {
     console.error(
       chalk.red(`Expected only a single action, found: \`${val}\``)
     );
-    actions.help.fn();
+    printUsage();
     process.exit(STATUS_BAD_PARAM);
   }
 
-  options.ACTION = actions[val];
+  currentAction = actions[val];
 
-  if (!options.ACTION) {
+  if (!currentAction) {
     console.error(chalk.red(`Invalid action \`${val}\``));
-    actions.help.fn();
+    printUsage();
     process.exit(STATUS_BAD_PARAM);
   }
 
-  if (options.ACTION.forwardsRestParams) {
+  if (currentAction.forwardsRestParams) {
     options.PARAMS = process.argv.slice(i + 1);
     break;
   }
 }
 
-if (!options.ACTION) {
+if (!currentAction) {
   console.error(chalk.red('No action provided'));
-  actions.help.fn();
+  printUsage();
   process.exit(STATUS_BAD_PARAM);
 }
 
-if (options.ACTION === actions.exec && options.PARAMS.length === 0) {
+if (currentAction === actions.exec && options.PARAMS.length === 0) {
   console.error(chalk.red('You should provide a command to exec'));
-  actions.help.fn();
+  printUsage(undefined, [currentAction.name]);
+  process.exit(STATUS_BAD_PARAM);
+}
+
+if (
+  ![actions.init, actions.install, actions.update].includes(currentAction) &&
+  options.DOCKER_IMAGE
+) {
+  console.error(chalk.red('Invalid flag: image'));
+  printUsage(undefined, [currentAction.name]);
+  process.exit(STATUS_BAD_PARAM);
+}
+
+if (
+  options.VENDOR_STRAT &&
+  !['submodule', 'subtree', 'manual'].includes(options.VENDOR_STRAT)
+) {
+  console.error(chalk.red(`Invalid strategy \`${options.VENDOR_STRAT}\``));
+  printUsage();
   process.exit(STATUS_BAD_PARAM);
 }
 
 readProjectInfo()
   .then((info) =>
-    options.ACTION.fn(
+    currentAction.fn(
       {
         ...info,
         options,
-        ...options.ACTION,
+        ...currentAction,
       },
-      options.PARAMS
+      options.PARAMS ?? []
     )
   )
   .catch((e) => {
+    if (e instanceof ParameterError) {
+      console.error(chalk.red(e.message));
+      printUsage(undefined, [currentAction.name]);
+      process.exit(STATUS_BAD_PARAM);
+    }
     const userTargetedError = e instanceof CommandError && e.userCommand;
 
     // Show additional information to user if verbose or we did a mistake
@@ -108,6 +149,10 @@ readProjectInfo()
     // We don't have a user targeted error anymore, we did a mistake for sure
     process.exit(STATUS_ERROR);
   })
+  .then(() => {
+    // Everything was done, update the configuration file if not exiting early
+    return writeProjectInfo();
+  })
   .finally(() => {
     process.exit(STATUS_OK);
   });

package/modules/actions/exec.js

@@ -0,0 +1,91 @@
+const path = require('path');
+
+const { CONTAINER_TARGET_PATH } = require('../constants');
+const { log, dockerExec, toPosixPath } = require('../helpers');
+
+const { fn: start } = require('./start');
+const { dockerHostUserParams, installDependencies } = require('./utils');
+
+function dockerRelativeWorkdir(libdragonInfo) {
+  return (
+    CONTAINER_TARGET_PATH +
+    '/' +
+    toPosixPath(path.relative(libdragonInfo.root, process.cwd()))
+  );
+}
+
+function dockerRelativeWorkdirParams(libdragonInfo) {
+  return ['--workdir', dockerRelativeWorkdir(libdragonInfo)];
+}
+
+const exec = async (libdragonInfo, commandAndParams) => {
+  log(
+    `Running ${commandAndParams[0]} at ${dockerRelativeWorkdir(
+      libdragonInfo
+    )} with [${commandAndParams.slice(1)}]`,
+    true
+  );
+
+  const tryCmd = (libdragonInfo) =>
+    libdragonInfo.containerId &&
+    dockerExec(
+      libdragonInfo,
+      [
+        ...dockerRelativeWorkdirParams(libdragonInfo),
+        ...dockerHostUserParams(libdragonInfo),
+      ],
+      commandAndParams,
+      true,
+      true // Cannot use "full" here, we need to know if the container is alive
+    );
+
+  let started = false;
+  const startOnceAndCmd = async () => {
+    if (!started) {
+      const newId = await start(libdragonInfo);
+      started = true;
+
+      // Re-install vendors on new container if one was created upon start
+      // Ideally we would want the consumer to handle dependencies and rebuild
+      // libdragon if necessary. Currently this saves the day with a little bit
+      // extra waiting when the container is deleted.
+      if (libdragonInfo.containerId !== newId) {
+        await installDependencies({
+          ...libdragonInfo,
+          containerId: newId,
+        });
+      }
+      await tryCmd({
+        ...libdragonInfo,
+        containerId: newId,
+      });
+      return newId;
+    }
+  };
+
+  if (!libdragonInfo.containerId) {
+    log(`Container does not exist for sure, restart`, true);
+    await startOnceAndCmd();
+    return;
+  }
+
+  try {
+    await tryCmd(libdragonInfo);
+  } catch (e) {
+    if (
+      !e.out ||
+      // TODO: is there a better way?
+      !e.out.toString().includes(libdragonInfo.containerId)
+    ) {
+      throw e;
+    }
+    await startOnceAndCmd();
+  }
+};
+
+module.exports = {
+  name: 'exec',
+  fn: exec,
+  forwardsRestParams: true,
+  showStatus: true,
+};

package/modules/actions/help.js

@@ -0,0 +1,157 @@
+const chalk = require('chalk');
+const commandLineUsage = require('command-line-usage');
+
+const { log } = require('../helpers');
+
+const printUsage = (_, actionArr) => {
+  const globalOptionDefinitions = [
+    {
+      name: 'verbose',
+      description:
+        'Be verbose. This will print all commands dispatched and their outputs as well. Will also start printing error stack traces.',
+      alias: 'v',
+      typeLabel: ' ',
+      group: 'global',
+    },
+  ];
+
+  const optionDefinitions = [
+    {
+      name: 'image',
+      description:
+        'Use this flag to provide a custom image to use instead of the default. It should include the toolchain at `/n64_toolchain`. It will cause a re-initialization of the container if a different image is provided.\n',
+      alias: 'i',
+      typeLabel: '<docker-image>',
+      group: 'docker',
+    },
+    {
+      name: 'directory',
+      description: `Directory where libdragon files are expected. It must be a project relative path as it will be mounted on the docker container. The cli will create and manage it when using a non-manual strategy. Defaults to \`./libdragon\` if not provided.\n`,
+      alias: 'd',
+      typeLabel: '<path>',
+      group: 'vendoring',
+    },
+    {
+      name: 'strategy',
+      description: `libdragon Vendoring strategy. Defaults to \`submodule\`, which safely creates a git repository at project root and a submodule at \`--directory\` to automatically update the vendored libdragon files.
+
+        With \`subtree\`, the cli will create a subtree at \`--directory\` instead. Keep in mind that git user name and email must be set up for this to work. Do not use if you are not using git yourself.
+
+        To disable auto-vendoring, init with \`manual\`. With \`manual\`, libdragon files are expected at the location provided by \`--directory\` flag and the user is responsible for vendoring and updating them. This will allow using any other manual vendoring method.
+
+        With the \`manual\` strategy, it is still recommended to have a git repository at project root such that container actions can execute faster by caching the container id inside the \`.git\` folder.
+
+        It is also possible to opt-out later on by running \`init\` with \`--strategy manual\`, though you will be responsible for managing the existing submodule/subtree.\n`,
+      alias: 'v',
+      typeLabel: '<strategy>',
+      group: 'vendoring',
+    },
+  ];
+
+  const actions = {
+    help: {
+      name: 'help [action]',
+      summary: 'Display this help information or details for the given action.',
+    },
+    init: {
+      name: 'init',
+      summary: 'Create a libdragon project in the current directory.',
+      description: `Creates a libdragon project in the current directory. Every libdragon project will have its own docker container instance. If you are in a git repository or an NPM project, libdragon will be initialized at their root also marking there with a \`.libdragon\` folder. Do not remove the \`.libdragon\` folder and commit its contents if you are using source control, as it keeps persistent libdragon project information.
+
+      By default, a git repository and a submodule at \`./libdragon\` will be created to automatically update the vendored libdragon files on subsequent \`update\`s. If you intend to opt-out from this feature, see the \`--strategy manual\` flag to provide your self-managed libdragon copy. The default behaviour is intended for users who primarily want to consume libdragon as is.
+
+      If this is the first time you are creating a libdragon project at that location, this action will also create skeleton project files to kickstart things with the given image, if provided. For subsequent runs, it will act like \`start\` thus can be used to revive an existing project without modifying it.`,
+      group: ['docker', 'vendoring'],
+    },
+    make: {
+      name: 'make [params]',
+      summary: 'Run the libdragon build system in the current directory.',
+      description: `Runs the libdragon build system in the current directory. It will mirror your current working directory to the container.
+
+      This action is a shortcut to the \`exec\` action under the hood.`,
+    },
+    exec: {
+      name: 'exec <command>',
+      summary: 'Execute given command in the current directory.',
+      description: `Executes the given command in the container passing down any arguments provided. If you change your host working directory, the command will be executed in the corresponding folder in the container as well.
+
+      This action will first try to execute the command in the container and if the container is not accessible, it will attempt a complete \`start\` cycle.
+
+      This will properly passthrough your TTY if you have one. So by running \`libdragon exec bash\` you can start an interactive bash session with full TTY support.`,
+    },
+    start: {
+      name: 'start',
+      summary: 'Start the container for current project.',
+      description:
+        'Start the container assigned to the current libdragon project. Will first attempt to start an existing container if found, followed by a new container run and installation similar to the `install` action. Will always print out the container id on success.',
+    },
+    name: {
+      name: 'stop',
+      summary: 'Stop the container for current project.',
+      description:
+        'Stop the container assigned to the current libdragon project.',
+    },
+    install: {
+      name: 'install',
+      summary: 'Vendor libdragon as is.',
+      group: ['docker'],
+      description: `Attempts to build and install everything libdragon related into the container. This includes all the tools and third parties used by libdragon except for the toolchain. If you have made changes to libdragon, you can execute this action to build everything based on your changes. Requires you to have an intact vendoring target (also see the \`--directory\` flag). If you are not working on libdragon itself, you can just use the \`update\` action instead.
+
+        This can be useful to recover from a half-baked container.`,
+    },
+    update: {
+      name: 'update',
+      summary: 'Update libdragon and do an install.',
+      description:
+        'If you are using auto-vendoring (see `--strategy`), this action will update the submodule/subtree from the remote branch (`trunk`) with a merge/squash strategy and then perform a `libdragon install`. This is the same as an `install` when the vendoring strategy is `manual`. You can use the `install` action to only update all libdragon related artifacts in the container.',
+      group: ['docker'],
+    },
+  };
+
+  const actionsToShow = actionArr
+    ?.filter((action) => Object.keys(actions).includes(action))
+    .filter((action) => !['help'].includes(action));
+
+  const sections = [
+    {
+      header: chalk.green('Usage:'),
+      content: 'libdragon [flags] <action>',
+    },
+    ...(actionsToShow?.length
+      ? actionsToShow.flatMap((action) => [
+          {
+            header: chalk.green(`${action} action:`),
+            content: actions[action].description,
+          },
+          actions[action].group
+            ? {
+                header: `accepted flags:`,
+                optionList: optionDefinitions,
+                group: actions[action].group,
+              }
+            : {},
+        ])
+      : [
+          {
+            header: chalk.green('Available Commands:'),
+            content: Object.values(actions).map((action) => ({
+              name: action.name,
+              summary: action.summary,
+            })),
+          },
+        ]),
+    {
+      header: chalk.green('Global flags:'),
+      optionList: globalOptionDefinitions,
+    },
+  ];
+  const usage = commandLineUsage(sections);
+  log(usage);
+};
+
+module.exports = {
+  name: 'help',
+  fn: printUsage,
+  showStatus: true,
+  forwardsRestParams: true,
+};