@npmcli/template-oss 2.9.2 → 3.1.1

package/README.md

@@ -7,106 +7,115 @@ single devDependency.
 
 ### Configuration
 
-Configure the use of `template-oss` in the root `package.json`.
+Configure the use of `@npmcli/template-oss` in your `package.json` using the
+`templateOSS` property.
+
 
 ```js
 {
   name: 'my-package',
-  // ...
   templateOSS: {
     // copy repo specific files for the root pkg
-    applyRootRepoFiles: true,
+    rootRepo: true,
     // modify package.json and copy module specific files for the root pkg
-    applyRootModuleFiles: true,
-    // copy repo files for each whitelisted workspaces
-    applyWorkspaceRepoFiles: true,
-    // whitelist workspace by package name to modify package.json
-    // and copy module files
+    rootModule: true,
+    // copy repo files for all workspaces
+    workspaceRepo: true,
+    // copy module files for all workspaces
+    workspaceModule: true,
+    // filter allowed workspaces by package name
+    // defaults to all workspaces
     workspaces: ['workspace-package-name'],
-    version: '2.3.1'
+    // The rest of the config is passed in as variables
+    // that can be used to template files in the content
+    // directory. Some common ones are:
+    // Turns off ci in windows
+    windowsCI: false,
+    // Change the versions tested in CI and engines
+    ciVersions: ['10', '12', '14']
   }
 }
+```
 
-### `package.json` patches
+#### Workspaces
 
-These fields will be set in the project's `package.json`:
+Individual workspaces can also supply their own config, if they are included by
+the root package's `templateOSS.workspaces` array. These settings will override
+any of the same settings in the root.
 
 ```js
 {
-  author: 'GitHub Inc.',
-  files: ['bin', 'lib'],
-  license: 'ISC',
-  templateVersion: $TEMPLATE_VERSION,
-  scripts: {
-    lint: `eslint '**/*.js'`,
-    postlint: 'npm-template-check',
-    lintfix: 'npm run lint -- --fix',
-    'template-copy': 'npm-template-copy --force',
-    preversion: 'npm test',
-    postversion: 'npm publish',
-    prepublishOnly: 'git push origin --follow-tags',
-    snap: 'tap',
-    test: 'tap',
-    posttest: 'npm run lint',
-  },
-  engines: {
-    node: '^12.13.0 || ^14.15.0 || >=16',
-  },
+  name: 'my-workspace',
+  templateOSS: {
+    // copy repo files for this workspace
+    workspaceRepo: true,
+    // copy module files for this workspace
+    moduleRepo: true,
+    // Changes windowsCI setting for this workspace
+    windowsCI: false,
+  }
 }
 ```
 
-The `"templateVersion"` field will be set to the version of this package being
-installed. This is used to determine if the postinstall script should take any
-action.
+### Content
+
+All the templated content for this repo lives in
+[`lib/content/`](./lib/content/). The `index.js`[./lib/content/index.js] file
+controls how and where this content is written.
+
+Content files can be overwritten or merged with the existing target file.
+Currently mergining is only supported for `package.json` files.
 
-#### Extending
+Each content file goes through the following pipeline:
 
-The `changes` constant located in `lib/postinstall/update-package.js` should contain
-all patches for the `package.json` file. Be sure to correctly expand any object/array
-based values with the original package content.
+1. It is read from its source location
+1. It is are templated using Handlebars with the variables from each packages's
+   config (with some derived values generated in [`config.js`](./lib/config.js)
+1. It is parsed based on its file extension in
+   [`parser.js`](./lib/util/parser.js)
+1. Additional logic is applied by the parser
+1. It is written to its target location
 
-### Static files
+### Usage
 
-Any existing `.eslintrc.*` files will be removed, unless they also match the
-pattern `.eslintrc.local.*`
+This package provides two bin scripts:
 
-These files will be copied, overwriting any existing files:
+#### `template-oss-check`
 
-- `.eslintrc.js`
-- `.github/workflows/ci.yml`
-- `.github/ISSUE_TEMPLATE/bug.yml`
-- `.github/ISSUE_TEMPLATE/config.yml`
-- `.github/CODEOWNERS`
-- `.gitignore`
-- `LICENSE.md`
-- `SECURITY.md`
+This will check if any of the applied files different from the target content,
+or if any of the other associated checks fail. The diffs of each file or check
+will be reported with instructions on how to fix it.
 
-### Dynamic Files
+#### `template-oss-apply [--force]`
 
-Currently, the only dynamic file generated is a github workflow for a given workspace.
-`.github/workflows/ci-$$package-name$$.yml`
+This will write all source files to their target locations in the cwd. It will
+do nothing if `package.json#templateOSS.version` is the same as the version
+being run. This can be overridden by `--force`.
 
-#### Extending
+This is the script that is run on `postinsall`.
 
-Place files in the `lib/content/` directory, use only the file name and remove
-any leading `.` characters (i.e. `.github/workflows/ci.yml` becomes `ci.yml`
-and `.gitignore` becomes `gitignore`).
+### Extending
 
-Modify the `repoFiles` and `moduleFiles` objects at the top of `lib/postinstall/copy-content.js` to include
-your new file. The object keys are destination paths, and values are source.
+#### `lib/apply`
 
-### `package.json` checks
+This directory is where all the logic for applying files lives. It should be
+possible to add new files without modifying anything in this directory. To add a
+file, add the templated file to `lib/content/$FILENAME` and add entry for it in
+`lib/content/index.js` depending on where and when it should be written (root vs
+workspace, repo vs module, add vs remove, etc).
 
-`npm-template-check` is run by `postlint` and will error if the `package.json`
-is not configured properly, with steps to run to correct any problems.
+#### `lib/check`
 
-### Manual copy
+All checks live in this directory and have the same signature. A check must be
+added to `lib/check/index.js` for it to be run.
 
-Template files will be copied automatically when `template-oss` is updated.
-You can force an update with `npm run template-copy`.
+#### Generic vs specific extensions
 
-#### Extending
+This repo is designed so that all (fine, most) of the logic in `lib/` is generic
+and could be applied across projects of many different types.
 
-Add any unwanted packages to `unwantedPackages` in `lib/check.js`. Currently
-the best way to install any packages is to include them as `peerDependencies`
-in this repo.
+The files in `lib/content` are extremely specific to the npm CLI. It would be
+trivial to swap out this content directory for a different one as it is only
+referenced in a single place in `lib/config.js`. However, it's not currently
+possible to change this value at runtime, but that might become possible in
+future versions of this package.

package/bin/apply.js

@@ -0,0 +1,22 @@
+#!/usr/bin/env node
+
+const apply = require('../lib/apply/index.js')
+
+const main = async () => {
+  const {
+    npm_config_global: globalMode,
+    npm_config_local_prefix: root,
+  } = process.env
+
+  // do nothing in global mode or when the local prefix isn't set
+  if (globalMode === 'true' || !root) {
+    return
+  }
+
+  await apply(root)
+}
+
+module.exports = main().catch((err) => {
+  console.error(err.stack)
+  process.exitCode = 1
+})

package/bin/check.js

@@ -0,0 +1,26 @@
+#!/usr/bin/env node
+
+const check = require('../lib/check/index.js')
+const output = require('../lib/util/output.js')
+
+const main = async () => {
+  const {
+    npm_config_local_prefix: root,
+  } = process.env
+
+  if (!root) {
+    throw new Error('This package requires npm >7.21.1')
+  }
+
+  const problems = await check(root)
+
+  if (problems.length) {
+    process.exitCode = 1
+    console.error(output(problems))
+  }
+}
+
+module.exports = main().catch((err) => {
+  console.error(err.stack)
+  process.exitCode = 1
+})

package/lib/apply/apply-files.js

@@ -0,0 +1,31 @@
+const fs = require('@npmcli/fs')
+const log = require('proc-log')
+const { rmEach, parseEach } = require('../util/files.js')
+
+const run = async (dir, files, options) => {
+  const { rm = [], add = {} } = files
+
+  log.verbose('apply-files', 'rm', rm)
+  await rmEach(dir, rm, options, (f) => fs.rm(f))
+
+  log.verbose('apply-files', 'add', add)
+  await parseEach(dir, add, options, (p) => p.applyWrite())
+}
+
+module.exports = [{
+  run: (options) => run(
+    options.config.repoDir,
+    options.config.repoFiles,
+    options
+  ),
+  when: ({ config: c }) => c.isForce || (c.needsUpdate && c.applyRepo),
+  name: 'apply-repo',
+}, {
+  run: (options) => run(
+    options.config.moduleDir,
+    options.config.moduleFiles,
+    options
+  ),
+  when: ({ config: c }) => c.isForce || (c.needsUpdate && c.applyModule),
+  name: 'apply-module',
+}]

package/lib/apply/index.js

@@ -0,0 +1,5 @@
+const run = require('../index.js')
+
+module.exports = (root, content) => run(root, content, [
+  require('./apply-files.js'),
+])

package/lib/check/check-apply.js

@@ -0,0 +1,73 @@
+const log = require('proc-log')
+const { relative, basename } = require('path')
+const { rmEach, parseEach } = require('../util/files.js')
+const { partition } = require('lodash')
+
+const solution = 'npx template-oss-apply --force'
+
+const run = async (type, dir, files, options) => {
+  const res = []
+  const rel = (f) => relative(options.root, f)
+  const { add: addFiles = {}, rm: rmFiles = [] } = files
+
+  const rm = await rmEach(dir, rmFiles, options, (f) => rel(f))
+  const [add, update] = partition(await parseEach(dir, addFiles, options, async (p) => {
+    const diff = await p.applyDiff()
+    const target = rel(p.target)
+    if (diff === null) {
+      // needs to be added
+      return target
+    } else if (diff === true) {
+      // its ok, no diff, this is filtered out
+      return null
+    }
+    return { file: target, diff }
+  }), (d) => typeof d === 'string')
+
+  log.verbose('check-apply', 'rm', rm)
+  if (rm.length) {
+    res.push({
+      title: `The following ${type} files need to be deleted:`,
+      body: rm,
+      solution,
+    })
+  }
+
+  log.verbose('check-apply', 'add', add)
+  if (add.length) {
+    res.push({
+      title: `The following ${type} files need to be added:`,
+      body: add,
+      solution,
+    })
+  }
+
+  log.verbose('check-apply', 'update', update)
+  res.push(...update.map(({ file, diff }) => ({
+    title: `The ${type} file ${basename(file)} needs to be updated:`,
+    body: [`${file}\n${'='.repeat(40)}\n${diff}`],
+    solution,
+  })))
+
+  return res
+}
+
+module.exports = [{
+  run: (options) => run(
+    'repo',
+    options.config.repoDir,
+    options.config.repoFiles,
+    options
+  ),
+  when: ({ config: c }) => c.applyRepo,
+  name: 'check-repo',
+}, {
+  run: (options) => run(
+    'module',
+    options.config.moduleDir,
+    options.config.moduleFiles,
+    options
+  ),
+  when: ({ config: c }) => c.applyModule,
+  name: 'check-module',
+}]

package/lib/check/check-changelog.js

@@ -0,0 +1,31 @@
+const fs = require('@npmcli/fs')
+const { EOL } = require('os')
+const { join, relative } = require('path')
+
+const run = async ({ root, path }) => {
+  // XXX: our changelogs are always markdown
+  // but they could be other extensions so
+  // make this glob for possible matches
+  const changelog = join(path, 'CHANGELOG.md')
+
+  if (await fs.exists(changelog)) {
+    const content = await fs.readFile(changelog, { encoding: 'utf8' })
+    const mustStart = `# Changelog${EOL}${EOL}#`
+    if (!content.startsWith(mustStart)) {
+      return {
+        title: `The ${relative(root, changelog)} is incorrect:`,
+        body: [
+          'The changelog should start with',
+          `"${mustStart}"`,
+        ],
+        solution: 'reformat the changelog to have the correct heading',
+      }
+    }
+  }
+}
+
+module.exports = {
+  run,
+  when: ({ config: c }) => c.applyModule,
+  name: 'check-changelog',
+}

package/lib/check/check-gitignore.js

@@ -0,0 +1,67 @@
+const log = require('proc-log')
+const { EOL } = require('os')
+const { resolve, relative, basename } = require('path')
+const fs = require('@npmcli/fs')
+const git = require('@npmcli/git')
+
+const NAME = 'check-gitignore'
+
+// The problem we are trying to solve is when a new .gitignore file
+// is copied into an existing repo, there could be files already checked in
+// to git that are now ignored by new gitignore rules. We want to warn
+// about these files.
+const run = async ({ root, path, config }) => {
+  log.verbose(NAME, { root, path })
+
+  const relativeToRoot = (f) => relative(root, resolve(path, f))
+
+  // use the root to detect a git repo but the project directory (path) for the
+  // ignore check
+  const ignoreFile = resolve(path, '.gitignore')
+  if (!await git.is({ cwd: root }) || !await fs.exists(ignoreFile)) {
+    log.verbose(NAME, 'no git or no gitignore')
+    return null
+  }
+
+  log.verbose(NAME, `using ignore file ${ignoreFile}`)
+
+  const res = await git.spawn([
+    'ls-files',
+    '--cached',
+    '--ignored',
+    // https://git-scm.com/docs/git-ls-files#_exclude_patterns
+    `--${config.isRoot ? 'exclude-from' : 'exclude-per-directory'}=${basename(ignoreFile)}`,
+  ], { cwd: path })
+
+  log.verbose(NAME, 'ls-files', res)
+
+  // TODO: files should be filtered if they have already been moved/deleted
+  // but not committed. Currently you must commit for this check to pass.
+  const files = res.stdout
+    .trim()
+    .split('\n')
+    .filter(Boolean)
+
+  if (!files.length) {
+    return null
+  }
+
+  const ignores = (await fs.readFile(ignoreFile))
+    .toString()
+    .split(EOL)
+    .filter((l) => l && !l.trim().startsWith('#'))
+
+  const relIgnore = relativeToRoot(ignoreFile)
+
+  return {
+    title: `The following files are tracked by git but matching a pattern in ${relIgnore}:`,
+    body: files.map(relativeToRoot),
+    solution: ['move files to not match one of the following patterns:', ...ignores],
+  }
+}
+
+module.exports = {
+  run,
+  when: ({ config: c }) => c.applyModule,
+  name: NAME,
+}

package/lib/check/check-required.js

@@ -0,0 +1,55 @@
+const log = require('proc-log')
+const npa = require('npm-package-arg')
+const { partition } = require('lodash')
+const hasPackage = require('../util/has-package.js')
+
+const rmCommand = (specs) =>
+  `npm rm ${specs.map((s) => s.name).join(' ')}`.trim()
+
+const installCommand = (specs, flags) => specs.length ?
+  `npm i ${specs.map((s) => `${s.name}@${s.fetchSpec}`).join(' ')} ${flags.join(' ')}`.trim() : ''
+
+// ensure required packages are present in the correct place
+const run = ({ pkg, path, config: { requiredPackages = {} } }) => {
+  // keys are the dependency location in package.json
+  // values are a filtered list of parsed specs that dont exist in the current package
+  // { [location]: [spec1, spec2] }
+  const requiredByLocation = Object.entries(requiredPackages)
+    .reduce((acc, [location, pkgs]) => {
+      acc[location] = pkgs
+        .filter((spec) => !hasPackage(pkg, spec, [location], path))
+        .map((spec) => npa(spec))
+      log.verbose(location, pkg, pkgs)
+      return acc
+    }, {})
+
+  const requiredEntries = Object.entries(requiredByLocation)
+
+  log.verbose('check-required', requiredEntries)
+
+  if (requiredEntries.flatMap(([, specs]) => specs).length) {
+    return requiredEntries.map(([location, specs]) => {
+      const locationFlag = hasPackage.flags[location]
+      const [exactSpecs, saveSpecs] = partition(specs, (s) => s.type === 'version')
+
+      log.verbose('check-required', location, specs)
+
+      return {
+        title: `The following required ${location} were not found:`,
+        body: specs.map((s) => s.rawSpec ? `${s.name}@${s.rawSpec}` : s.name),
+        // solution is to remove any existing all at once but add back in by --save-<location>
+        solution: [
+          rmCommand(specs),
+          installCommand(saveSpecs, [locationFlag]),
+          installCommand(exactSpecs, [locationFlag, '--save-exact']),
+        ].filter(Boolean).join(' && '),
+      }
+    })
+  }
+}
+
+module.exports = {
+  run,
+  when: ({ config: c }) => c.applyModule,
+  name: 'check-required-packages',
+}

package/lib/check/check-unwanted.js

@@ -0,0 +1,23 @@
+
+const hasPackage = require('../util/has-package.js')
+
+const run = ({ pkg, config: { allowedPackages = [], unwantedPackages = [] } }) => {
+  // ensure packages that should not be present are removed
+  const hasUnwanted = unwantedPackages
+    .filter((name) => !allowedPackages.includes(name))
+    .filter((name) => hasPackage(pkg, name))
+
+  if (hasUnwanted.length) {
+    return {
+      title: 'The following unwanted packages were found:',
+      body: hasUnwanted,
+      solution: `npm rm ${hasUnwanted.join(' ')}`,
+    }
+  }
+}
+
+module.exports = {
+  run,
+  when: ({ config: c }) => c.applyModule,
+  name: 'check-unwanted-packages',
+}

package/lib/check/index.js

@@ -0,0 +1,9 @@
+const run = require('../index.js')
+
+module.exports = (root, content) => run(root, content, [
+  require('./check-apply.js'),
+  require('./check-required.js'),
+  require('./check-unwanted.js'),
+  require('./check-gitignore.js'),
+  require('./check-changelog.js'),
+])