aberlaas 2.0.0 → 2.2.0

package/README.md

@@ -51,7 +51,6 @@ The following table lists all the scripts added:
 
 | Script                     | Description                                               |
 | -------------------------- | --------------------------------------------------------- |
-| `yarn run husky:precommit` | Run before any commit (through Husky)                     |
 | `yarn run test`            | Run tests using Jest                                      |
 | `yarn run test:watch`      | Run tests using Jest in watch mode                        |
 | `yarn run ci`              | Run testing and linting in CI                             |
@@ -111,7 +110,7 @@ describe('moduleName', () => {
 
 ## Precommit hooks
 
-`aberlaas` uses `husky` and `lint-staged` to make sure all committed code
+`aberlaas` uses `lint-staged` to make sure all committed code
 follows your coding standard.
 
 All `css`, `js`, `json` and `yml` files will be checked for parsing errors
@@ -189,10 +188,133 @@ exported by the package and thus can be `require`d in userland.
 extends the configuration exported in the previous files. Copying files to
 userland allows user to change the files if they want to change the behavior.
 
-`.eslintrc.js`, `.stylelintrc.js`, `jest.config.js` and `.huskyrc.js` are local
+`.eslintrc.js`, `.stylelintrc.js` and `jest.config.js` are local
 configuration files for `aberlaas` itself. They eat their own dog food by
 referencing the same configs as above.
 
+## Tools used and their future
+
+### ESLint
+
+ESLint doesn't yet support ESM config files. We'll upgrade to the latest ESLint
+when it does. This should also help fix the issue with Yarn PnP (see below).
+
+### Yarn
+
+**tl;dr; We'll move to Yarn PnP once ESLint has support for Flat Configs, and
+default `yarn run` doesn't add a ~1s delay overhead**
+
+#### PnP
+
+
+Aberlaas is using Yarn Berry (v2+) as its main package management tool.
+
+Yarn Berry comes with a Plug And Play (PnP) feature that replaces the usage of
+`node_modules` in favor of a `.pnp.cjs` file. Instead of having a very large
+`node_modules` folder, a unique copy of each dependency is stored in the user
+home folder and the `.pnp.cjs` file only keeps references to those folder. This
+makes installing dependencies faster as it needs way less I/O.
+
+By ditching the whole `node_modules` principle, it also removes concepts like
+hoisting of dependencies in a monorepo. This, unfortunately, breaks ESLint.
+
+ESLint expect all its plugins to be defined as `peerDependencies` at the root of
+a mono-repo, as it will always try to include them from there. It works more or
+less correctly at the best of times, and `aberlaas` already has some hacks
+(including `resolvePluginsRelativeTo`) to work around that.
+
+But with PnP, there is no way to make it work correctly, so I will need to wait
+for a better compatibility between ESLint and Yarn 2 before using it.
+
+Sources:
+- [#8](https://github.com/yarnpkg/berry/issues/8), the initial case study of Yarn + ESLint
+- [GitHub issue that explicitly explain the problem](https://github.com/yarnpkg/berry/discussions/3909)
+- [`eslint-patch`, an ESLint plugin that hacks around this issue](https://yarnpkg.com/package?name=@rushstack/eslint-patch)
+- [The `resolvePluginsRelativeTo` bandaid from ESLint](https://eslint.org/docs/latest/use/command-line-interface#--resolve-plugins-relative-to)
+- [A nice postmortem of moving to Yarn 2](https://www.dolthub.com/blog/2022-03-18-migrating-to-yarn-2/)
+- [The Flat Config feature in ESLint that should solve the problem](https://eslint.org/docs/latest/use/configure/configuration-files-new)
+
+#### Calling binaries from the host
+
+In yarn v1, if you install `aberlaas` in your project, all of `aberlaas`
+dependencies (including binaries) were hoisted to the root. This was a design
+flaw, as if several dependencies defined binaries by the same name, they would
+fight in a race condition to take the slot in `node_modules/.bin`.
+
+This has been fixed in Yarn Berry, but it also means it's no longer possible to
+call `yarn run eslint` from a repository that includes `aberlaas`, because
+`eslint` is not a direct dependency of the repo.
+
+It might be possible to define proxy binaries inside of `aberlaas`, but those
+will incur performance issues as they will need to spawn one more yarn context
+(see below).
+
+#### Calling binaries from aberlaas
+
+In Yarn V1, I was calling binaries that `aberlaas` depends on (`eslint`, `vitest`,
+etc) by calling `yarn run`. This was adding some overhead but was acceptable.
+Yarn Berry adds even more overhead and it is becoming noticeable.
+
+Now, my prefered way it to use the NodeJS API of the dependencies instead of
+their CLI, to limit the overhead. I managed to move most tools to this new
+approach, but sometimes I still need to use the CLI (`vitest` for example has
+awesome live watching and display reloading that I don't think I can easily
+replicate through code).
+
+I ran some performance tests to see what would be the fastest way to call
+`vitest` from `aberlaas`
+
+```
+hyperfine \
+    "zsh -i -c 'yarn bin vitest && /home/tim/local/www/projects/aberlaas/node_modules/vitest/vitest.mjs --version'" \
+    "zsh -i -c 'yarn run vitest --version'" \
+    "/home/tim/local/www/projects/aberlaas/node_modules/vitest/vitest.mjs --version" \
+    "/home/tim/local/www/projects/aberlaas/node_modules/.bin/vitest --version" 
+Benchmark 1: zsh -i -c 'yarn run vitest --version'
+  Time (mean ± σ):      1.945 s ±  0.051 s    [User: 1.986 s, System: 0.850 s]
+  Range (min … max):    1.859 s …  2.018 s    10 runs
+ 
+Benchmark 2: zsh -i -c 'yarn bin vitest && /home/tim/local/www/projects/aberlaas/node_modules/vitest/vitest.mjs --version'
+  Time (mean ± σ):      2.108 s ±  0.150 s    [User: 2.108 s, System: 0.843 s]
+  Range (min … max):    1.930 s …  2.289 s    10 runs
+ 
+Benchmark 3: /home/tim/local/www/projects/aberlaas/node_modules/vitest/vitest.m
+js --version
+  Time (mean ± σ):     482.5 ms ±  40.9 ms    [User: 448.4 ms, System: 327.2 ms]
+  Range (min … max):   442.1 ms … 553.3 ms    10 runs
+ 
+Benchmark 4: /home/tim/local/www/projects/aberlaas/node_modules/.bin/vitest --version
+  Time (mean ± σ):     491.9 ms ±  29.6 ms    [User: 454.1 ms, System: 331.2 ms]
+  Range (min … max):   453.8 ms … 535.4 ms    10 runs
+```
+
+Finding the binary through `yarn bin` then calling it is the slowest, but `yarn
+run` isn't much faster. Directly calling the binary is the fastest, but it's
+path can't be easily guessed (apart from reading and parsing a `package.json`).
+But using the symlinks in `node_modules/.bin` is consistent, barely slower than
+calling the binary directly and much faster than using yarn.
+
+This is what I'll be using. Of course, this will break when I'll move to PnP as
+the `node_modules` folder won't be there, but hopefully Yarn will be faster by
+then and I can use `yarn run` reliably.
+
+#### Speed
+
+With Yarn 2+, calling `yarn run` seem to add ~1s of overhead each time. This is
+a known issue (due to the fact Yarn 2 needs to spawn yarn 1, node and a few
+other layers). 
+
+1s is not much in itself, but grows quickly when you have nested `yarn run
+aberlaas` calls that call `yarn run eslint`, and even more when you need to deal
+with monorepos and multiple packages that each have their own yarn scope.
+
+There are open issues on the topic, but nothing merged yet:
+
+- [#3732](https://github.com/yarnpkg/berry/issues/3732), where it is discussed to make yarn run aware that it's running yarn run
+  and keep the same state without spawning new yarns
+- [#2575](https://github.com/yarnpkg/berry/issues/2575), which is the main issue about Yarn performance, with benchmarks against
+  npm/npx/yarn v1.
+
 ## Where does the name Aberlaas come from?
 
 Aberlaas is the base camp from which all great expedition start in the _La Horde
@@ -206,4 +328,4 @@ For your convenience, `aberlass` and `aberlas` are added as aliases by default.
 
 ## Documentation
 
-The complete documentation can be found on https://projects.pixelastic.com/aberlaas/
+The complete documentation can be found on https://projects.pixelastic.com/aberlaas/

package/bin/{aberlaas → aberlaas.js}

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-const aberlaas = require('../main.js');
+import aberlaas from '../main.js';
 (async () => {
   await aberlaas.run(process.argv.slice(2));
 })();

package/commands/ci/autoRelease.js

@@ -1,11 +1,12 @@
-const release = require('../release.js');
-const consoleInfo = require('firost/consoleInfo');
-const _ = require('golgoth/lodash');
-const run = require('firost/run');
-const write = require('firost/write');
-const exists = require('firost/exists');
-const helper = require('../../helper.js');
-module.exports = {
+import release from '../release/index.js';
+import consoleInfo from 'firost/consoleInfo.js';
+import _ from 'golgoth/lodash.js';
+import run from 'firost/run.js';
+import write from 'firost/write.js';
+import exists from 'firost/exists.js';
+import helper from '../../helper.js';
+
+export default {
   /**
    * Run a git command in the repo
    *

package/commands/ci/index.js

@@ -1,13 +1,13 @@
-const helper = require('../../helper.js');
-const ciInfo = require('ci-info');
-const _ = require('golgoth/lodash');
-const pMap = require('golgoth/pMap');
-const readJson = require('firost/readJson');
-const run = require('firost/run');
-const consoleInfo = require('firost/consoleInfo');
-const autoRelease = require('./autoRelease.js');
+import helper from '../../helper.js';
+import ciInfo from 'ci-info';
+import _ from 'golgoth/lodash.js';
+import pMap from 'golgoth/pMap.js';
+import readJson from 'firost/readJson.js';
+import run from 'firost/run.js';
+import consoleInfo from 'firost/consoleInfo.js';
+import autoRelease from './autoRelease.js';
 
-module.exports = {
+export default {
   /**
    * Return the value of an environment variable
    * @param {string} key Name of the variable
@@ -112,7 +112,7 @@ module.exports = {
 
         await helper.yarnRun(command);
       },
-      { concurrency: 1 }
+      { concurrency: 1 },
     );
 
     // Attempt to release the package if --auto-release is set

package/commands/compress/dummy.js

@@ -0,0 +1,11 @@
+/**
+ * This is a temporary, dummy file to be able to test the top-level
+ * compress.run() function. I assume I'll add more compression types in addition
+ * to png, so this dummy compress is just to test that all compress are
+ * correctly called, but it doesn't do anything
+ **/
+export default {
+  async run() {
+    return true;
+  },
+};

package/commands/compress/index.js

@@ -1,12 +1,14 @@
-const _ = require('golgoth/lodash');
-const pMap = require('golgoth/pMap');
-const consoleError = require('firost/consoleError');
-const firostError = require('firost/error');
-const helper = require('../../helper.js');
+import _ from 'golgoth/lodash.js';
+import pMap from 'golgoth/pMap.js';
+import consoleError from 'firost/consoleError.js';
+import firostError from 'firost/error.js';
+import compressPng from './png.js';
+import compressDummy from './dummy.js';
 
-module.exports = {
+export default {
   types: {
-    png: './png.js',
+    png: compressPng,
+    dummy: compressDummy,
   },
   /**
    * Wrapper to compress all supported formats
@@ -22,7 +24,7 @@ module.exports = {
     await pMap(typesToCompress, async (type) => {
       try {
         const userPatterns = _.get(cliArgs, '_');
-        const compresser = require(this.types[type]);
+        const compresser = this.types[type];
 
         await compresser.run(userPatterns);
       } catch (error) {
@@ -37,16 +39,5 @@ module.exports = {
 
     return true;
   },
-  /**
-   * Find all relevant files of the specified extension in the host
-   * Note: Should be used by child classes
-   * @param {Array} safeListExtension List of allowed extensions to keep
-   * @param {Array} userPatterns Patterns to narrow the search down
-   * @returns {Array} Array of files
-   **/
-  async getInputFiles(safeListExtension, userPatterns = []) {
-    const inputPatterns = _.isEmpty(userPatterns) ? '.' : userPatterns;
-    return await helper.findHostFiles(inputPatterns, safeListExtension);
-  },
   __consoleError: consoleError,
 };

package/commands/compress/png.js

@@ -1,24 +1,29 @@
-const lint = require('./index.js');
-const run = require('firost/run');
-const which = require('firost/which');
-const firostError = require('firost/error');
+import helper from '../../helper.js';
+import _ from 'golgoth/lodash.js';
+import run from 'firost/run.js';
+import which from 'firost/which.js';
+import firostError from 'firost/error.js';
 
-module.exports = {
+export default {
   /**
    * Find the png files to compress
    * @param {Array} userPatterns Patterns to narrow the search down
    * @returns {Array} Array of files
    **/
   async getInputFiles(userPatterns) {
-    return await lint.getInputFiles(['.png'], userPatterns);
+    const filePatterns = _.isEmpty(userPatterns)
+      ? ['./**/*.png']
+      : userPatterns;
+    return await helper.findHostFiles(filePatterns, ['.png']);
   },
+
   /**
-   * Check if the binary is available in the $PATH
-   * @returns {boolean} True if available, false otherwise
+   
+   * Returns path to the binary to execute
+   * @returns {string|boolean} Path to the binary, or false if not found
    **/
-  async hasBin() {
-    const binary = await which('pngmin');
-    return !!binary;
+  async getBinaryPath() {
+    return await this.__which('pngmin');
   },
   /**
    * Compress files
@@ -27,16 +32,20 @@ module.exports = {
    **/
   async run(userPatterns) {
     // Stop early if no bin
-    if (!(await this.hasBin())) {
+    const binaryPath = await this.getBinaryPath();
+    if (!binaryPath) {
       return true;
     }
 
     try {
       const files = await this.getInputFiles(userPatterns);
-      const command = `pngmin ${files.join(' ')}`;
+      const command = `${binaryPath} ${files.join(' ')}`;
       await run(command, { stdout: false });
     } catch (error) {
       throw firostError('PngCompressError', error.message);
     }
+
+    return true;
   },
+  __which: which,
 };

package/commands/{init.js → init/index.js}

@@ -1,19 +1,21 @@
-const _ = require('golgoth/lodash');
-const pMapSeries = require('golgoth/pMapSeries');
-const isFile = require('firost/isFile');
-const copy = require('firost/copy');
-const readJson = require('firost/readJson');
-const writeJson = require('firost/writeJson');
-const exists = require('firost/exists');
-const read = require('firost/read');
-const write = require('firost/write');
-const run = require('firost/run');
-const spinner = require('firost/spinner');
-const helper = require('../helper');
-const path = require('path');
-const nodeConfig = require('../configs/node');
-const consoleInfo = require('firost/consoleInfo');
-module.exports = {
+import _ from 'golgoth/lodash.js';
+import pMapSeries from 'golgoth/pMapSeries.js';
+import isFile from 'firost/isFile.js';
+import copy from 'firost/copy.js';
+import readJson from 'firost/readJson.js';
+import writeJson from 'firost/writeJson.js';
+import exists from 'firost/exists.js';
+import read from 'firost/read.js';
+import write from 'firost/write.js';
+import run from 'firost/run.js';
+import spinner from 'firost/spinner.js';
+import helper from '../../helper.js';
+import path from 'path';
+import nodeConfig from '../../configs/node.cjs';
+import consoleInfo from 'firost/consoleInfo.js';
+import Gilmore from 'gilmore';
+
+export default {
   /**
    * Copy a config template to the host
    * @param {string} source Path to source file, relative to aberlaas
@@ -43,35 +45,35 @@ module.exports = {
    * default rules and overwrite them as they see fit
    **/
   async addConfigFiles() {
+    // Yarn
+    await this.copyToHost('templates/_yarnrc.yml', '.yarnrc.yml');
+
     // ESLint
-    await this.copyToHost('templates/_eslintrc.js', '.eslintrc.js');
+    await this.copyToHost('templates/_eslintrc.cjs', '.eslintrc.cjs');
     await this.copyToHost('templates/_eslintignore.conf', '.eslintignore');
 
-    // Husky
-    await this.copyToHost('templates/_huskyrc.js', '.huskyrc.js');
-
     // Lint-staged
-    await this.copyToHost('templates/_lintstagedrc.js', '.lintstagedrc.js');
+    await this.copyToHost('templates/_lintstagedrc.cjs', '.lintstagedrc.cjs');
 
-    // Jest
-    await this.copyToHost('templates/jest.config.js', 'jest.config.js');
+    // Vite
+    await this.copyToHost('templates/vite.config.js', 'vite.config.js');
 
     // Prettier
-    await this.copyToHost('templates/_prettierrc.js', '.prettierrc.js');
+    await this.copyToHost('templates/_prettierrc.cjs', '.prettierrc.cjs');
 
     // Stylelint
-    await this.copyToHost('templates/_stylelintrc.js', '.stylelintrc.js');
+    await this.copyToHost('templates/_stylelintrc.cjs', '.stylelintrc.cjs');
 
     // Renovate
     await this.copyToHost(
       'templates/_github/renovate.json',
-      '.github/renovate.json'
+      '.github/renovate.json',
     );
 
     // CircleCI
     await this.copyToHost(
       'templates/_circleci/config.yml',
-      '.circleci/config.yml'
+      '.circleci/config.yml',
     );
   },
   /**
@@ -95,7 +97,7 @@ module.exports = {
     const newPackage = _.set(
       _.clone(currentPackage),
       `scripts.${scriptName}`,
-      `./${scriptPath}`
+      `./${scriptPath}`,
     );
     await writeJson(newPackage, packagePath);
     return true;
@@ -107,9 +109,9 @@ module.exports = {
   async addScripts() {
     const defaultScripts = [
       { key: 'ci', value: 'scripts/ci' },
+      { key: 'compress', value: 'scripts/compress' },
       { key: 'lint', value: 'scripts/lint' },
       { key: 'lint:fix', value: 'scripts/lint-fix' },
-      { key: 'husky:precommit', value: 'scripts/husky-precommit' },
       { key: 'release', value: 'scripts/release' },
       { key: 'test', value: 'scripts/test' },
       { key: 'test:watch', value: 'scripts/test-watch' },
@@ -122,6 +124,7 @@ module.exports = {
   /**
    * Update package.json with .main and .files keys to use files in ./lib
    * directory by default
+   * TODO: Update this to use exports instead of main
    **/
   async setDefaultReleaseFiles() {
     const packagePath = helper.hostPath('package.json');
@@ -135,7 +138,7 @@ module.exports = {
       await this.copyToHost('templates/lib/main.js', 'lib/main.js');
       await this.copyToHost(
         'templates/lib/__tests__/main.js',
-        'lib/__tests__/main.js'
+        'lib/__tests__/main.js',
       );
     }
 
@@ -251,6 +254,20 @@ module.exports = {
     // Download latest yarn version
     await this.__run('yarn set version', { stdout: false });
   },
+  /**
+   * Configure git hooks to use scripts/hooks instead of .git/hooks
+   **/
+  async configureGitHooks() {
+    // Copy the scripts into ./scripts/hooks
+    await this.copyToHost(
+      './templates/scripts/hooks/pre-commit',
+      './scripts/hooks/pre-commit',
+    );
+
+    // Configure git to use this directory for hooks
+    const repo = new Gilmore(helper.hostRoot());
+    await repo.setConfig('core.hooksPath', 'scripts/hooks');
+  },
   /**
    * Copy all config files and configure the scripts
    **/
@@ -267,6 +284,9 @@ module.exports = {
     progress.tick('Adding yarn scripts');
     await this.addScripts();
 
+    progress.tick('Configuring git hooks');
+    await this.configureGitHooks();
+
     progress.tick('Updating LICENSE');
     await this.addLicenseFile();
     await this.addLicenseField();
@@ -283,7 +303,7 @@ module.exports = {
     progress.success('aberlaas project initialized');
 
     this.__consoleInfo(
-      "Don't forget to run aberlaas setup after pushing your repository"
+      "Don't forget to run aberlaas setup after pushing your repository",
     );
   },
   __run: run,

package/commands/lint/circleci.js

@@ -1,20 +1,19 @@
-const helper = require('../../helper.js');
-const lint = require('./index.js');
-const lintYml = require('./yml.js');
-const ciInfo = require('ci-info');
-const exists = require('firost/exists');
-const run = require('firost/run');
-const which = require('firost/which');
-const firostError = require('firost/error');
+import helper from '../../helper.js';
+import lintYml from './yml.js';
+import ciInfo from 'ci-info';
+import exists from 'firost/exists.js';
+import run from 'firost/run.js';
+import which from 'firost/which.js';
+import firostError from 'firost/error.js';
 
-module.exports = {
+export default {
   configPath: '.circleci/config.yml',
   /**
-   * Find the CircleCI config file
+   * Find all relevant files
    * @returns {Array} Array of files
    **/
   async getInputFiles() {
-    return await lint.getInputFiles(['.yml'], [this.configPath]);
+    return await helper.findHostFiles([this.configPath]);
   },
   /**
    * Check if the code is currently running on CircleCI

package/commands/lint/css.js

@@ -1,46 +1,55 @@
-const helper = require('../../helper.js');
-const lint = require('./index.js');
-const _ = require('golgoth/lodash');
-const firostError = require('firost/error');
-const run = require('firost/run');
-module.exports = {
+import helper from '../../helper.js';
+import { fix as prettierFix } from './helpers/prettier.js';
+import stylelint from 'stylelint';
+import _ from 'golgoth/lodash.js';
+import firostError from 'firost/error.js';
+
+export default {
   /**
    * Find all relevant files
    * @param {Array} userPatterns Patterns to narrow the search down
    * @returns {Array} Array of files
    **/
   async getInputFiles(userPatterns) {
-    return await lint.getInputFiles(['.css'], userPatterns);
+    const filePatterns = _.isEmpty(userPatterns)
+      ? ['./**/*.css']
+      : userPatterns;
+    return await helper.findHostFiles(filePatterns, ['.css']);
   },
   /**
    * Lint all files and display results.
    * @param {Array} userPatterns Patterns to narrow the search down
    * @param {string} userConfigFile Custom config file to use
+   * @param {object} userOptions Options to pass to ESLint, including fix
    * @returns {boolean} True on success
    **/
-  async run(userPatterns, userConfigFile) {
+  async run(userPatterns, userConfigFile, userOptions = {}) {
+    // Options
+    const options = { fix: false, ...userOptions };
+
+    // Files
     const files = await this.getInputFiles(userPatterns);
     if (_.isEmpty(files)) {
       return true;
     }
 
+    // Config
     const configFile = await helper.configFile(
       userConfigFile,
-      '.stylelintrc.js',
-      'configs/stylelint.js'
+      '.stylelintrc.cjs',
+      'configs/stylelint.cjs',
     );
-    const binary = await helper.which('stylelint');
-    const options = [...files, '--color', '--config', configFile];
-    try {
-      await run(`${binary} ${options.join(' ')}`, { stdout: false });
-    } catch (error) {
-      // If it fails because no files passed actually exists, it's not really
-      // a failure
-      const errorMessage = error.stdout;
-      if (_.startsWith(errorMessage, 'Error: No files matching the pattern')) {
-        return true;
-      }
-      throw firostError('ERROR_CSS_LINT', error.stdout);
+    const config = await helper.import(configFile);
+
+    const result = await stylelint.lint({
+      config,
+      files,
+      formatter: 'string',
+      ...options,
+    });
+
+    if (result.errored) {
+      throw firostError('ERROR_CSS_LINT', result.output);
     }
     return true;
   },
@@ -56,10 +65,9 @@ module.exports = {
       return true;
     }
     // Try to pretiffy as much as we can
-    await lint.fixWithPrettier(files);
+    await prettierFix(files);
     // Still run a lint on it so it can fail if not everything is fixed
-    await this.run(userPatterns, userConfigFile);
+    await this.run(userPatterns, userConfigFile, { fix: true });
     return true;
   },
-  __run: run,
 };