libdragon 10.3.1 → 10.4.0

package/CHANGELOG.md

@@ -1,5 +1,41 @@
 # Change Log
 
+## [10.4.0] - 2022-03-23
+
+### Fixed
+
+- Update the root makefile to utilize `SOURCE_DIR` for example builds. Then we are
+  able to map container files to local files properly with a generic regex in the
+  problem matcher. This fixes #13 and does not change any behaviour.
+- Add missing examples to the vscode run configurations.
+- Install and build libdragon related things in the container when `exec` and
+  `make` causes a new container run. This was previously prevented on `v10.3.1`
+  because it was unnecessarily delaying all exec operations when the container
+  is started. Refactoring things allowed me to realize this can be improved
+  instead of forcing the user to do a manual `install`.
+- Fix a potential issue that may cause the git commands to run in current folder
+  instead of the project root.
+- Attach the error handler once for spawnProcess.
+- Update vulnerable dependencies.
+
+### Added
+
+- `--directory` option to customize vendoring location.
+- `--strategy` option to select a vendoring strategy. Currently supported options
+  are `submodule`, `subtree` and `manual`. The default is `submodule` and `manual`
+  can be used to opt-out of auto vendoring. Useful if the user wants to utilize
+  a different vendoring strategy and opt-out of the auto-managed git flows.
+
+### Changed
+
+- Migrate to a json file for persistent project information.
+- Only save the configuration file on successful exit except for the initial
+  migration.
+- Do not prevent init if there is a file named libdragon in the target folder.
+  This used to cause problems on windows but I cannot reproduce it anymore
+  with `2.33.1.windows.1`. It may be something caused by my old configuration.
+- Minor performance improvements.
+
 ## [10.3.1] - 2022-01-25
 
 ### Fixed
@@ -19,7 +55,7 @@
 
 ### Changed
 
-- Only accept he image flag for init, install, and update actions as documented.
+- Only accept the image flag for init, install, and update actions as documented.
 - Improve documentation for the `init` and `install` actions.
 - Do not attempt an `install` when running `exec`, just start the container. If
   there is a half-baked container, a manual `install` will potentially restore it.

package/README.md

@@ -22,7 +22,7 @@ To update the library and rebuild/install all the artifacts;
 
     libdragon update
 
-See [next section](#overall-usage) for more details.
+`git` is not strictly required to use this tool as docker's git will be used instead. Still, it is highly recommended to have git on your host machine.
 
 ## Overall usage
 
@@ -38,16 +38,6 @@ You can invoke libdragon as follows;
 
 Run `libdragon help [action]` for more details on individual actions.
 
-### Available flags
-
-**`--image <docker-image>`**
-
-Use this flag to provide a custom image to use instead of the default. It should include the toolchain at `/n64_toolchain`. It will be effective for `init`, `install` and `update` actions and will cause a re-initialization of the container if an image different from what was written to project configuration is provided.
-
-**`--verbose`**
-
-Be verbose. This will print all commands dispatched and their outputs as well.
-
 ## Working on this repository
 
 After cloning this repository on a system with node.js (`>= 14`) & docker (`>= 18`), in this repository's root do;
@@ -92,7 +82,7 @@ To update the submodule and re-build everything;
 
 ### Local test bench
 
-This repository also uses [ed64](https://github.com/anacierdem/ed64), so you can just hit F5 on vscode (The `Run Test Bench` launch configuration) to run the test code in `src` folder to develop libdragon itself quicker if you have an everdrive. There is a caveat though: If you want the problem matcher to work properly, you should name this repository folder `libdragon` exactly.
+This repository also uses [ed64](https://github.com/anacierdem/ed64), so you can just hit F5 on vscode (The `Run Test Bench` launch configuration) to run the test code in `src` folder to develop libdragon itself quicker if you have an everdrive.
 
 There are also additional vscode launch configurations to build libdragon examples and tests based on the currently built and installed libdragon in the docker container. Most of these will always rebuild so that they will use the latest if you made and installed an alternative libdragon. The test bench itself and a few examples (that use the new build system) will already rebuild and reinstall libdragon automatically. These will always produce a rom image using the latest libdragon code in the active repository via its make dependencies. You can clean everything with the `clean` task (open the command palette and choose `Run Task -> clean`).
 

package/index.js

@@ -1,13 +1,17 @@
 #!/usr/bin/env node
 
 const chalk = require('chalk');
-const { readProjectInfo, CommandError, globals } = require('./modules/helpers');
-const actions = require('./modules/actions');
-const { printUsage } = require('./modules/usage');
 
-const STATUS_OK = 0;
-const STATUS_ERROR = 1;
-const STATUS_BAD_PARAM = 2;
+const actions = require('./modules/actions');
+const { fn: printUsage } = require('./modules/actions/help');
+const {
+  STATUS_OK,
+  STATUS_BAD_PARAM,
+  STATUS_ERROR,
+} = require('./modules/constants');
+const { globals } = require('./modules/globals');
+const { CommandError, ParameterError } = require('./modules/helpers');
+const { readProjectInfo, writeProjectInfo } = require('./modules/project-info');
 
 let options = {},
   currentAction;
@@ -32,6 +36,22 @@ for (let i = 2; i < process.argv.length; i++) {
     continue;
   }
 
+  if (['--directory', '-d'].includes(val)) {
+    options.VENDOR_DIR = process.argv[++i];
+    continue;
+  } else if (val.indexOf('--directory=') === 0) {
+    options.VENDOR_DIR = val.split('=')[1];
+    continue;
+  }
+
+  if (['--strategy', '-s'].includes(val)) {
+    options.VENDOR_STRAT = process.argv[++i];
+    continue;
+  } else if (val.indexOf('--strategy=') === 0) {
+    options.VENDOR_STRAT = val.split('=')[1];
+    continue;
+  }
+
   if (val.indexOf('--') >= 0) {
     console.error(chalk.red(`Invalid flag \`${val}\``));
     printUsage();
@@ -81,6 +101,15 @@ if (
   process.exit(STATUS_BAD_PARAM);
 }
 
+if (
+  options.VENDOR_STRAT &&
+  !['submodule', 'subtree', 'manual'].includes(options.VENDOR_STRAT)
+) {
+  console.error(chalk.red(`Invalid strategy \`${options.VENDOR_STRAT}\``));
+  printUsage();
+  process.exit(STATUS_BAD_PARAM);
+}
+
 readProjectInfo()
   .then((info) =>
     currentAction.fn(
@@ -93,6 +122,11 @@ readProjectInfo()
     )
   )
   .catch((e) => {
+    if (e instanceof ParameterError) {
+      console.error(chalk.red(e.message));
+      printUsage(undefined, [currentAction.name]);
+      process.exit(STATUS_BAD_PARAM);
+    }
     const userTargetedError = e instanceof CommandError && e.userCommand;
 
     // Show additional information to user if verbose or we did a mistake
@@ -115,6 +149,10 @@ readProjectInfo()
     // We don't have a user targeted error anymore, we did a mistake for sure
     process.exit(STATUS_ERROR);
   })
+  .then(() => {
+    // Everything was done, update the configuration file if not exiting early
+    return writeProjectInfo();
+  })
   .finally(() => {
     process.exit(STATUS_OK);
   });

package/modules/actions/exec.js

@@ -0,0 +1,91 @@
+const path = require('path');
+
+const { CONTAINER_TARGET_PATH } = require('../constants');
+const { log, dockerExec, toPosixPath } = require('../helpers');
+
+const { fn: start } = require('./start');
+const { dockerHostUserParams, installDependencies } = require('./utils');
+
+function dockerRelativeWorkdir(libdragonInfo) {
+  return (
+    CONTAINER_TARGET_PATH +
+    '/' +
+    toPosixPath(path.relative(libdragonInfo.root, process.cwd()))
+  );
+}
+
+function dockerRelativeWorkdirParams(libdragonInfo) {
+  return ['--workdir', dockerRelativeWorkdir(libdragonInfo)];
+}
+
+const exec = async (libdragonInfo, commandAndParams) => {
+  log(
+    `Running ${commandAndParams[0]} at ${dockerRelativeWorkdir(
+      libdragonInfo
+    )} with [${commandAndParams.slice(1)}]`,
+    true
+  );
+
+  const tryCmd = (libdragonInfo) =>
+    libdragonInfo.containerId &&
+    dockerExec(
+      libdragonInfo,
+      [
+        ...dockerRelativeWorkdirParams(libdragonInfo),
+        ...dockerHostUserParams(libdragonInfo),
+      ],
+      commandAndParams,
+      true,
+      true // Cannot use "full" here, we need to know if the container is alive
+    );
+
+  let started = false;
+  const startOnceAndCmd = async () => {
+    if (!started) {
+      const newId = await start(libdragonInfo);
+      started = true;
+
+      // Re-install vendors on new container if one was created upon start
+      // Ideally we would want the consumer to handle dependencies and rebuild
+      // libdragon if necessary. Currently this saves the day with a little bit
+      // extra waiting when the container is deleted.
+      if (libdragonInfo.containerId !== newId) {
+        await installDependencies({
+          ...libdragonInfo,
+          containerId: newId,
+        });
+      }
+      await tryCmd({
+        ...libdragonInfo,
+        containerId: newId,
+      });
+      return newId;
+    }
+  };
+
+  if (!libdragonInfo.containerId) {
+    log(`Container does not exist for sure, restart`, true);
+    await startOnceAndCmd();
+    return;
+  }
+
+  try {
+    await tryCmd(libdragonInfo);
+  } catch (e) {
+    if (
+      !e.out ||
+      // TODO: is there a better way?
+      !e.out.toString().includes(libdragonInfo.containerId)
+    ) {
+      throw e;
+    }
+    await startOnceAndCmd();
+  }
+};
+
+module.exports = {
+  name: 'exec',
+  fn: exec,
+  forwardsRestParams: true,
+  showStatus: true,
+};

package/modules/{usage.js → actions/help.js}

@@ -1,14 +1,16 @@
 const chalk = require('chalk');
 const commandLineUsage = require('command-line-usage');
 
-const { log } = require('./helpers');
+const { log } = require('../helpers');
 
 const printUsage = (_, actionArr) => {
   const globalOptionDefinitions = [
     {
       name: 'verbose',
-      description: 'Be verbose',
+      description:
+        'Be verbose. This will print all commands dispatched and their outputs as well. Will also start printing error stack traces.',
       alias: 'v',
+      typeLabel: ' ',
       group: 'global',
     },
   ];
@@ -16,11 +18,34 @@ const printUsage = (_, actionArr) => {
   const optionDefinitions = [
     {
       name: 'image',
-      description: 'Provide a custom image.',
+      description:
+        'Use this flag to provide a custom image to use instead of the default. It should include the toolchain at `/n64_toolchain`. It will cause a re-initialization of the container if a different image is provided.\n',
       alias: 'i',
       typeLabel: '<docker-image>',
       group: 'docker',
     },
+    {
+      name: 'directory',
+      description: `Directory where libdragon files are expected. It must be a project relative path as it will be mounted on the docker container. The cli will create and manage it when using a non-manual strategy. Defaults to \`./libdragon\` if not provided.\n`,
+      alias: 'd',
+      typeLabel: '<path>',
+      group: 'vendoring',
+    },
+    {
+      name: 'strategy',
+      description: `libdragon Vendoring strategy. Defaults to \`submodule\`, which safely creates a git repository at project root and a submodule at \`--directory\` to automatically update the vendored libdragon files.
+
+        With \`subtree\`, the cli will create a subtree at \`--directory\` instead. Keep in mind that git user name and email must be set up for this to work. Do not use if you are not using git yourself.
+
+        To disable auto-vendoring, init with \`manual\`. With \`manual\`, libdragon files are expected at the location provided by \`--directory\` flag and the user is responsible for vendoring and updating them. This will allow using any other manual vendoring method.
+
+        With the \`manual\` strategy, it is still recommended to have a git repository at project root such that container actions can execute faster by caching the container id inside the \`.git\` folder.
+
+        It is also possible to opt-out later on by running \`init\` with \`--strategy manual\`, though you will be responsible for managing the existing submodule/subtree.\n`,
+      alias: 'v',
+      typeLabel: '<strategy>',
+      group: 'vendoring',
+    },
   ];
 
   const actions = {
@@ -31,12 +56,12 @@ const printUsage = (_, actionArr) => {
     init: {
       name: 'init',
       summary: 'Create a libdragon project in the current directory.',
-      description: `Creates a libdragon project in the current directory. Every libdragon project will have its own docker container instance. If you are in a git repository or an NPM project, libdragon will be initialized at their root also marking there with a \`.libdragon\` folder.
+      description: `Creates a libdragon project in the current directory. Every libdragon project will have its own docker container instance. If you are in a git repository or an NPM project, libdragon will be initialized at their root also marking there with a \`.libdragon\` folder. Do not remove the \`.libdragon\` folder and commit its contents if you are using source control, as it keeps persistent libdragon project information.
 
-      A git repository and a submodule at \`./libdragon\` will also be created. Do not remove the \`.libdragon\` folder and commit its contents if you are using git, as it keeps persistent libdragon project information.
+      By default, a git repository and a submodule at \`./libdragon\` will be created to automatically update the vendored libdragon files on subsequent \`update\`s. If you intend to opt-out from this feature, see the \`--strategy manual\` flag to provide your self-managed libdragon copy. The default behaviour is intended for users who primarily want to consume libdragon as is.
 
       If this is the first time you are creating a libdragon project at that location, this action will also create skeleton project files to kickstart things with the given image, if provided. For subsequent runs, it will act like \`start\` thus can be used to revive an existing project without modifying it.`,
-      group: ['docker'],
+      group: ['docker', 'vendoring'],
     },
     make: {
       name: 'make [params]',
@@ -70,7 +95,7 @@ const printUsage = (_, actionArr) => {
       name: 'install',
       summary: 'Vendor libdragon as is.',
       group: ['docker'],
-      description: `Attempts to build and install everything libdragon related into the container. This includes all the tools and third parties used by libdragon except for the toolchain. If you have made changes to libdragon, you can execute this action to build everything based on your changes. Requires you to have an intact \`libdragon\` at the root of the project. If you are not working on libdragon, you can just use the \`update\` action instead.
+      description: `Attempts to build and install everything libdragon related into the container. This includes all the tools and third parties used by libdragon except for the toolchain. If you have made changes to libdragon, you can execute this action to build everything based on your changes. Requires you to have an intact vendoring target (also see the \`--directory\` flag). If you are not working on libdragon itself, you can just use the \`update\` action instead.
 
         This can be useful to recover from a half-baked container.`,
     },
@@ -78,7 +103,7 @@ const printUsage = (_, actionArr) => {
       name: 'update',
       summary: 'Update libdragon and do an install.',
       description:
-        'This action will update the submodule from the remote branch (`trunk`) with a merge strategy and then perform a `libdragon install`. You can use the `install` action to only update all libdragon related artifacts in the container.',
+        'If you are using auto-vendoring (see `--strategy`), this action will update the submodule/subtree from the remote branch (`trunk`) with a merge/squash strategy and then perform a `libdragon install`. This is the same as an `install` when the vendoring strategy is `manual`. You can use the `install` action to only update all libdragon related artifacts in the container.',
       group: ['docker'],
     },
   };
@@ -125,5 +150,8 @@ const printUsage = (_, actionArr) => {
 };
 
 module.exports = {
-  printUsage,
+  name: 'help',
+  fn: printUsage,
+  showStatus: true,
+  forwardsRestParams: true,
 };

package/modules/actions/index.js

@@ -0,0 +1,43 @@
+const { spawnProcess } = require('../helpers');
+const { checkContainerRunning } = require('./utils');
+
+const { fn: exec } = require('./exec');
+
+const stop = async (libdragonInfo) => {
+  const running =
+    libdragonInfo.containerId &&
+    (await checkContainerRunning(libdragonInfo.containerId));
+  if (!running) {
+    return;
+  }
+
+  await spawnProcess('docker', ['container', 'stop', running]);
+};
+
+const make = async (libdragonInfo, params) => {
+  await exec(libdragonInfo, ['make', ...params]);
+};
+
+// TODO: separate into files
+module.exports = {
+  start: require('./start'),
+  stop: {
+    name: 'stop',
+    fn: stop,
+    showStatus: false, // This will only print out the id
+  },
+  init: require('./init'),
+
+  exec: require('./exec'),
+  make: {
+    name: 'make',
+    fn: make,
+    forwardsRestParams: true,
+    showStatus: true,
+  },
+
+  install: require('./install'),
+  update: require('./update'),
+
+  help: require('./help'),
+};

package/modules/actions/init.js

@@ -0,0 +1,190 @@
+const fs = require('fs/promises');
+const path = require('path');
+
+const chalk = require('chalk');
+
+const { fn: install } = require('./install');
+const { fn: start } = require('./start');
+const {
+  installDependencies,
+  tryCacheContainerId,
+  updateImage,
+  runGitMaybeHost,
+} = require('./utils');
+
+const {
+  LIBDRAGON_PROJECT_MANIFEST,
+  LIBDRAGON_SUBMODULE,
+  LIBDRAGON_BRANCH,
+  LIBDRAGON_GIT,
+} = require('../constants');
+const {
+  log,
+  copyDirContents,
+  CommandError,
+  ParameterError,
+} = require('../helpers');
+const { setProjectInfoToSave } = require('../project-info');
+
+const autoVendor = async (libdragonInfo) => {
+  await runGitMaybeHost(libdragonInfo, ['init']);
+
+  if (libdragonInfo.vendorStrategy === 'submodule') {
+    await runGitMaybeHost(libdragonInfo, [
+      'submodule',
+      'add',
+      '--force',
+      '--name',
+      LIBDRAGON_SUBMODULE,
+      '--branch',
+      LIBDRAGON_BRANCH,
+      LIBDRAGON_GIT,
+      libdragonInfo.vendorDirectory,
+    ]);
+  } else if (libdragonInfo.vendorStrategy === 'subtree') {
+    // Create a commit if it does not exist. This is required for subtree.
+    try {
+      await runGitMaybeHost(libdragonInfo, ['rev-parse', 'HEAD']);
+    } catch (e) {
+      if (!(e instanceof CommandError)) throw e;
+
+      // This will throw if git user name/email is not set up. Let's not assume
+      // anything for now. This means subtree is not supported for someone without
+      // git on the host machine.
+      await runGitMaybeHost(libdragonInfo, [
+        'commit',
+        '--allow-empty',
+        '-n',
+        '-m',
+        'Initial commit.',
+      ]);
+    }
+
+    await runGitMaybeHost(libdragonInfo, [
+      'subtree',
+      'add',
+      '--prefix',
+      path.relative(libdragonInfo.root, libdragonInfo.vendorDirectory),
+      LIBDRAGON_GIT,
+      LIBDRAGON_BRANCH,
+      '--squash',
+    ]);
+  }
+
+  return libdragonInfo;
+};
+
+/**
+ * Initialize a new libdragon project in current working directory
+ * Also downloads the image
+ */
+async function init(libdragonInfo) {
+  log(`Initializing a libdragon project at ${libdragonInfo.root}`);
+
+  // TODO: use exists instead & check if it is a directory
+  const files = await fs.readdir(libdragonInfo.root);
+
+  const manifestFile = files.find(
+    (name) => name === LIBDRAGON_PROJECT_MANIFEST
+  );
+
+  let newInfo = libdragonInfo;
+
+  // Validate vendoring strategy. Do not allow a switch after successful initialization
+  if (
+    manifestFile &&
+    newInfo.options.VENDOR_STRAT &&
+    newInfo.options.VENDOR_STRAT !== 'manual' &&
+    newInfo.vendorStrategy !== newInfo.options.VENDOR_STRAT
+  ) {
+    throw new Error(
+      `Requested strategy switch: ${newInfo.vendorStrategy} -> ${newInfo.options.VENDOR_STRAT} It is not possible to switch vendoring strategy after initializing a project. You can always switch to manual and handle libdragon yourself.`
+    );
+  }
+
+  // Update the strategy information for the project if the flag is provided
+  if (newInfo.options.VENDOR_STRAT) {
+    newInfo = setProjectInfoToSave({
+      ...newInfo,
+      vendorStrategy: newInfo.options.VENDOR_STRAT,
+    });
+  }
+
+  // Update the directory information for the project if the flag is provided
+  if (newInfo.options.VENDOR_DIR) {
+    // Validate vendoring path
+    if (path.isAbsolute(newInfo.options.VENDOR_DIR)) {
+      throw new ParameterError(
+        '`--directory` must be project-relative as it will be mounted to the docker container.'
+      );
+    }
+
+    newInfo = setProjectInfoToSave({
+      ...newInfo,
+      vendorDirectory: newInfo.options.VENDOR_DIR,
+    });
+  }
+
+  if (manifestFile) {
+    log(
+      `${path.join(
+        newInfo.root,
+        manifestFile
+      )} exists. This is already a libdragon project, starting it...`
+    );
+    if (newInfo.options.DOCKER_IMAGE) {
+      log(
+        `Not changing docker image. Use the install action if you want to override the image.`
+      );
+    }
+    await install(newInfo);
+    return;
+  }
+
+  newInfo.imageName =
+    (await updateImage(newInfo, newInfo.imageName)) || newInfo.imageName;
+  // Download image and start it
+  const containerReadyPromise = start(newInfo).then((newId) => ({
+    ...newInfo,
+    containerId: newId,
+  }));
+
+  let vendorAndGitReadyPromise = containerReadyPromise;
+  if (newInfo.vendorStrategy !== 'manual') {
+    const libdragonFile = files.find((name) =>
+      name.match(
+        new RegExp(`^${path.relative(newInfo.root, newInfo.vendorDirectory)}$`)
+      )
+    );
+
+    if (libdragonFile) {
+      throw new Error(
+        `${path.join(
+          newInfo.root,
+          libdragonFile
+        )} already exists. That is the libdragon vendoring target, please remove and retry.`
+      );
+    }
+
+    vendorAndGitReadyPromise = containerReadyPromise.then(autoVendor);
+  }
+
+  log(`Preparing project files...`);
+  const skeletonFolder = path.join(__dirname, '../../skeleton');
+
+  await Promise.all([
+    // We have created a new container, save the new info
+    vendorAndGitReadyPromise.then(tryCacheContainerId),
+    vendorAndGitReadyPromise.then(installDependencies),
+    // node copy functions does not work with pkg
+    copyDirContents(skeletonFolder, newInfo.root),
+  ]);
+
+  log(chalk.green(`libdragon ready at \`${newInfo.root}\`.`));
+}
+
+module.exports = {
+  name: 'init',
+  fn: init,
+  showStatus: true,
+};

package/modules/actions/install.js

@@ -0,0 +1,52 @@
+const { log } = require('../helpers');
+const {
+  destroyContainer,
+  installDependencies,
+  updateImage,
+} = require('./utils');
+const { fn: start } = require('./start');
+
+/**
+ * Updates the image if flag is provided and install vendors onto the container.
+ * We should probably remove the image installation responsibility from this
+ * action but it might be a breaking change. Maybe we can keep it backward
+ * compatible with additional flags.
+ * @param libdragonInfo
+ */
+const install = async (libdragonInfo) => {
+  let containerId;
+  const oldImageName = libdragonInfo.imageName;
+  const imageName = libdragonInfo.options.DOCKER_IMAGE;
+  // If an image is provided, always attempt to install it
+  // See https://github.com/anacierdem/libdragon-docker/issues/47
+  if (imageName) {
+    log(`Changing image from \`${oldImageName}\` to \`${imageName}\``);
+
+    // Download the new image and if it is different, re-create the container
+    if (await updateImage(libdragonInfo, imageName)) {
+      await destroyContainer(libdragonInfo);
+    }
+
+    containerId = await start({
+      ...libdragonInfo,
+      imageName,
+    });
+  } else {
+    // Make sure existing one is running
+    containerId = await start(libdragonInfo);
+  }
+
+  // Re-install vendors on new image
+  // TODO: skip this if unnecessary
+  await installDependencies({
+    ...libdragonInfo,
+    imageName,
+    containerId,
+  });
+};
+
+module.exports = {
+  name: 'install',
+  fn: install,
+  showStatus: true,
+};