isolate-package 1.7.1 → 1.8.0-2

package/README.md

@@ -2,21 +2,18 @@
 
 <!-- TOC -->
 
+- [TLDR](#tldr)
 - [Introduction](#introduction)
 - [Features](#features)
 - [Motivation](#motivation)
-- [Install](#install)
-- [Usage as binary](#usage-as-binary)
-- [Usage as function](#usage-as-function)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Troubleshooting](#troubleshooting)
 - [Prerequisites](#prerequisites)
   - [Define shared dependencies in the package manifest](#define-shared-dependencies-in-the-package-manifest)
   - [Define "version" field in each package manifest](#define-version-field-in-each-package-manifest)
   - [Define "files" field in each package manifest](#define-files-field-in-each-package-manifest)
   - [Use a flat structure inside your packages folders](#use-a-flat-structure-inside-your-packages-folders)
-- [Working with Firebase](#working-with-firebase)
-  - [A Quick Start](#a-quick-start)
-  - [Deploying from multiple packages](#deploying-from-multiple-packages)
-  - [Deploying from the root](#deploying-from-the-root)
 - [Configuration Options](#configuration-options)
   - [buildDirName](#builddirname)
   - [excludeLockfile](#excludelockfile)
@@ -27,36 +24,38 @@
   - [tsconfigPath](#tsconfigpath)
   - [workspacePackages](#workspacepackages)
   - [workspaceRoot](#workspaceroot)
-- [Troubleshooting](#troubleshooting)
 - [Lockfiles](#lockfiles)
-  - [NPM and PNPM](#npm-and-pnpm)
-  - [Yarn](#yarn)
-  - [A Partial Workaround](#a-partial-workaround)
-- [Different Package Managers](#different-package-managers)
-- [Using the Firebase Functions Emulator](#using-the-firebase-functions-emulator)
+- [API](#api)
 - [The internal packages strategy](#the-internal-packages-strategy)
+- [Working with Firebase](#working-with-firebase)
+  - [A Quick Start](#a-quick-start)
+  - [Deploying from multiple packages](#deploying-from-multiple-packages)
+  - [Deploying from the root](#deploying-from-the-root)
+  - [Using the Firebase Functions Emulator](#using-the-firebase-functions-emulator)
 
 <!-- /TOC -->
 
+## TLDR
+
+Run `npx isolate-package isolate` from the monorepo package you would like to
+isolate.
+
 ## Introduction
 
 Isolate a monorepo workspace package to form a self-contained deployable package
 that includes internal dependencies and a compatible lockfile. The internal
-packages structure is preserved, so dependencies are not bundled.
+packages structure is preserved. Code is not bundled.
 
 ## Features
 
-- Isolate a monorepo package with its internal dependencies to form a
-  self-contained installable package.
-- Deterministic deployment by generating an isolated lockfile based on the
-  existing monorepo lockfile. Currently this feature is only supported for NPM
-  and PNPM. See [lockfiles](#lockfiles) for more information.
+- Isolates a monorepo package with its internal dependencies to form a
+  self-contained deployable directory.
+- Generates an isolated / pruned lockfile based on the existing monorepo
+  lockfile. See [lockfiles](#lockfiles) for more information.
 - Zero-config for the vast majority of use-cases, with no manual steps involved.
 - Support for PNPM, NPM and Yarn.
-- Compatible with the Firebase tools CLI, incl 1st gen and 2nd gen Firebase
-  functions deployments.
-- Uses a pack/unpack approach to isolate only those files that would have been
-  part of a published NPM package.
+- Compatible with the Firebase tools CLI, incl 1st and 2nd gen Firebase
+  functions.
 - Isolates internal workspace dependencies recursively. If package A depends on
   internal package B which depends on internal package C, all of them will be
   included.
@@ -65,7 +64,7 @@ packages structure is preserved, so dependencies are not bundled.
 ## Motivation
 
 This solution was born from a desire to deploy to
-[Firebase](https://firebase.google.com/) from a monorepo without requiring
+[Firebase](https://firebase.google.com/) from a monorepo without resorting to
 custom shell scripts and other hacks. Here is
 [an article](https://thijs-koerselman.medium.com/deploy-to-firebase-without-the-hacks-e685de39025e)
 explaining the issue in more detail.
@@ -76,20 +75,19 @@ related to Firebase.
 
 > !! There is now
 > [a fork of firebase-tools](https://github.com/0x80/firebase-tools-with-isolate),
-> where isolate-package is integrated. It is preferred because it simplifies the
-> setup and allows the isolation to run only as part of the deploy process,
+> where isolate-package is integrated. This is preferred because it simplifies
+> the setup and allows the isolation to run only as part of the deploy process,
 > preserving live code updates when running the local Firebase emulators.
 
-## Install
+## Installation
 
-Run `pnpm install isolate-package --dev` or the equivalent for `yarn` or `npm`.
+Run `pnpm install isolate-package --dev` or the equivalent for `npm` or `yarn`.
 
 I recommend using `pnpm` for
-[a number of reasons](https://pnpm.io/feature-comparison). Also, at the time of
-writing it is the only package manager for which isolate-package can generate a
-lockfile. For more information see [lockfiles](#lockfiles).
+[a number of reasons](https://pnpm.io/feature-comparison). In my experience it
+is the best package manager, especially for monorepo setups.
 
-## Usage as binary
+## Usage
 
 This package exposes a binary called `isolate`.
 
@@ -105,21 +103,18 @@ By default the isolated output will become available at `./isolate`.
 If you are here to simplify and improve your Firebase deployments check out the
 [Firebase quick start guide](#a-quick-start).
 
-## Usage as function
+## Troubleshooting
 
-Alternatively, `isolate` can be integrated in other programs by importing it as
-a function. You optionally pass it a some user configuration and possibly a
-logger to handle any output messages should you need to write them to a
-different location as the standard `node:console`.
+If something is not working as expected, add a `isolate.config.json` file, and
+set `"logLevel"` to `"debug"`. This should give you detailed feedback in the
+console.
 
-```ts
-import { isolate } from "isolate-package";
+In addition define an environment variable to debug the configuration being used
+by setting `DEBUG_ISOLATE_CONFIG=true` before you execute `isolate`.
 
-await isolate({
-  config: { logLevel: "debug" },
-  logger: customLogger,
-});
-```
+When debugging Firebase deployment issues it might be convenient to trigger the
+isolate process manually with `npx isolate` and possibly
+`DEBUG_ISOLATE_CONFIG=true npx isolate`
 
 If you do not pass in any configuration, the function will try to read a
 `isolate.config.json` file from disk. You can set
@@ -163,10 +158,9 @@ generate part of the packed filename. A personal preference is to set it to
 
 ### Define "files" field in each package manifest
 
-> NOTE: This step is not required if you use the [internal packages > > > > > >
->
-> > strategy](#the-internal-packages-strategy but you could set it to `["src"]`
-> > instead of `["dist"]`.
+> NOTE: This step is not required if you use the
+> [internal packages strategy](#the-internal-packages-strategy) but you could
+> set it to `["src"]` instead of `["dist"]`.
 
 The isolate process uses (p)npm `pack` to extract files from package
 directories, just like publishing a package would.
@@ -207,112 +201,6 @@ You can, however, declare multiple workspace packages directories. Personally, I
 prefer to use `["packages/*", "apps/*", "services/*"]`. It is only the structure
 inside them that should be flat.
 
-## Working with Firebase
-
-> !! There is now
-> [a fork of firebase-tools](https://github.com/0x80/firebase-tools-with-isolate),
-> where isolate-package is integrated. It is preferred because it simplifies the
-> setup and allows the isolation to run only as part of the deploy process,
-> preserving live code updates when running the local Firebase emulators.
-
-### A Quick Start
-
-If you are not confident that your monorepo setup is solid, please check out my
-in-dept example at [mono-ts](https://github.com/0x80/mono-ts) where many
-different aspects are discussed and `isolate-package` is used to demonstrate
-Firebase deployments.
-
-This section describes the steps required for Firebase deployment, assuming:
-
-- You use a fairly typical monorepo setup
-- Your `firebase.json` config lives in the root of the package that you like to
-  deploy to Firebase, hereafter referred to as the "target package".
-
-If your setup diverges from a traditional one, please continue reading the
-[Prerequisites](#prerequisites) section.
-
-1. In the target package, install `isolate-package` and `firebase-tools` by
-   running `pnpm add isolate-package firebase-tools -D` or the Yarn / NPM
-   equivalent. I tend to install firebase-tools as a devDependency in every
-   Firebase package, but you could also use a global install if you prefer that.
-2. In the `firebase.json` config set `"source"` to `"./isolate"` and
-   `"predeploy"` to `["turbo build", "isolate"]` or whatever suits your build
-   tool. The important part here is that isolate is being executed after the
-   build stage.
-3. From the target package folder, you should now be able to deploy with
-   `npx firebase deploy`.
-
-I recommend keeping a `firebase.json` file inside each Firebase package (as
-opposed to the monorepo root), because it allows you to deploy from multiple
-independent packages. It makes it easy to deploy 1st gen functions next to 2nd
-gen functions, deploy different node versions, and decrease the built output
-size and dependency lists for each package, improving deployment and cold-start
-times.
-
-### Deploying from multiple packages
-
-You can deploy to Firebase from multiple packages in your monorepo, in which
-case you co-locate your `firebase.json` file with the source code, and not in
-the root of the monorepo. If you do want to keep the firebase config in the
-root, read the instructions for
-[deploying to Firebase from the root](#deploying-to-firebase-from-the-root).
-
-In order to deploy to Firebase, the `functions.source` setting in
-`firebase.json` needs to point to the isolated output folder, which would be
-`./isolate` when using the default configuration.
-
-The `predeploy` phase should first build and then isolate the output.
-
-Here's an example using [Turborepo](https://turbo.build/):
-
-```cjson
-// firebase.json
-{
-  "functions": {
-    "source": "./isolate",
-    "predeploy": ["turbo build", "isolate"]
-  }
-}
-```
-
-With this configuration you can then run `npx firebase deploy --only functions`
-from the package.
-
-If you like to deploy to Firebase Functions from multiple packages you will also
-need to configure a unique `codebase` identifier for each of them. For more
-information,
-[read this](https://firebase.google.com/docs/functions/beta/organize-functions).
-
-Make sure your Firebase package adheres to the things mentioned in
-[prerequisites](#prerequisites) and its package manifest contains the field
-`"main"`, or `"module"` if you set `"type": "module"`, so Firebase knows the
-entry point to your source code.
-
-### Deploying from the root
-
-If, for some reason, you choose to keep the `firebase.json` file in the root of
-the monorepo you will have to place a configuration file called
-`isolate.config.json` in the root with the following content:
-
-```cjson
-// isolate.config.json
-{
-  "targetPackagePath": "./packages/your-firebase-package"
-}
-```
-
-The Firebase configuration should then look something like this:
-
-```cjson
-// firebase.json
-{
-  "functions": {
-    "source": "./packages/your-firebase-package/isolate",
-    "predeploy": ["turbo build", "isolate"]
-  }
-}
-```
-
 ## Configuration Options
 
 For most users no configuration should be necessary.
@@ -338,13 +226,16 @@ setting to specify where the build output files are located.
 
 Type: `boolean`, default: Depends on package manager.
 
+**Deprecated** This option exists from the time that lockfiles were not
+supported for all package managers. You should not need this escape hatch
+anymore.
+
 Sets the inclusion or exclusion of the lockfile as part of the deployment.
 
-Isolated NPM and PNPM lockfiles are generated based on the existing root
-lockfile, and they are included by default.
+Isolated / pruned lockfiles are generated for NPM, PNPM and Yarn v1 (classic)
+based on the existing root lockfile, and they are included by default.
 
-The lockfile for Yarn is excluded by default. For more information see
-[lockfiles](#lockfiles).
+For more information see [lockfiles](#lockfiles).
 
 ### includeDevDependencies
 
@@ -430,83 +321,191 @@ services
 
 When you use the `targetPackagePath` option, this setting will be ignored.
 
-## Troubleshooting
+## Lockfiles
 
-If something is not working as expected, add a `isolate.config.json` file, and
-set `"logLevel"` to `"debug"`. This should give you detailed feedback in the
-console.
+A lockfile in a monorepo describes the dependencies of all packages, and does
+not translate to the isolated output without altering it.
 
-In addition define an environment variable to debug the configuration being used
-by setting `DEBUG_ISOLATE_CONFIG=true` before you execute `isolate`.
+If you copying the original lockfile and deploy it with the isolated code, a CI
+environment will not accept the lockfile. It is also not possibly to generate a
+brand new lockfile from the isolated code by mimicking a fresh install, because
+versions would be able to diverge and thus negate the whole point of having a
+lockfile in the first place.
 
-When debugging Firebase deployment issues it might be convenient to trigger the
-isolate process manually with `npx isolate` and possibly
-`DEBUG_ISOLATE_CONFIG=true npx isolate`
+For this to work, we need to re-generate or prune the original lockfile to keep
+the versions of the original but only describe the dependencies of the code in
+the isolated output.
 
-## Lockfiles
+Since every package manager works completely different in this regard, this part
+of the puzzle had to be solved in a different ways for each of them, and not all
+are supported yet.
 
-A lockfile in a monorepo describes the dependencies of all packages, and does
-not necessarily translate to the isolated output without altering it. Different
-package managers use very different formats, and it might not be enough to do a
-find/replace on some paths.
+At the moment lockfiles for the following package managers are supported:
 
-It is also not possibly to generate a brand new lockfile from the isolated code
-by mimicking a fresh install, because versions would be able to diverge and thus
-it would negate the whole point of having a lockfile in the first place.
+- NPM
+- PNPM
+- Yarn Classic (v1)
 
-What we need is to re-generate a lockfile for the isolated output based on the
-versions that are currently installed and locked in the monorepo lockfile.
+More detailed information on the implementation can be found in the
+[additional docs](./docs/lockfiles.md).
 
-### NPM and PNPM
+## API
 
-For NPM and PNPM a lockfile is generated and included in the isolated output.
+Alternatively, `isolate` can be integrated in other programs by importing it as
+a function. You optionally pass it a some user configuration and possibly a
+logger to handle any output messages should you need to write them to a
+different location as the standard `node:console`.
 
-@TODO write about how it works an how we got there.
+```ts
+import { isolate } from "isolate-package";
 
-If you do somehow run into a problem related to the lockfile, you can opt-out of
-this by setting `excludeLockfile: true` in the `isolate.config.json`
-configuration file.
+await isolate({
+  config: { logLevel: "debug" },
+  logger: customLogger,
+});
+```
 
-### Yarn
+## The internal packages strategy
+
+An alternative approach to using internal dependencies in a Typescript monorepo
+is
+[the internal packages strategy](https://turbo.build/blog/you-might-not-need-typescript-project-references),
+in which the package manifest entries point directly to Typescript source files,
+to omit intermediate build steps. The approach is compatible with
+isolate-package and showcased in
+[my example monorepo setup](https://github.com/0x80/mono-ts)
+
+In summary this is how it works:
 
-For now, Yarn lockfiles are simply copied over to the isolated output. I believe
-I have seen Firebase deployments work with it, but it is likely you will run
-into an error.
+1. The package to be deployed lists its internal dependencies as usual, but the
+   package manifests of those dependencies point directly to the Typescript
+   source (and types).
+2. You configure the bundler of your target package to include the source code
+   for those internal packages in its output bundle. In the case of TSUP for the
+   [API service in the mono-ts](https://github.com/0x80/mono-ts/blob/main/services/api/tsup.config.ts)
+   that configuration is: `noExternal: ["@mono/common"]`
+3. When `isolate` runs, it does the same thing as always. It detects the
+   internal packages, copies them to the isolate output folder and adjusts any
+   links.
+4. When deploying to Firebase, the cloud pipeline will treat the package
+   manifest as usual, which installs the listed dependencies and any
+   dependencies listed in the linked internal package manifests.
+
+Steps 3 and 4 are no different from a traditional setup.
+
+Note that the manifests for the internal packages in the output will still point
+to the Typescript source files, but since the shared code was embedded in the
+bundle, they will never be referenced via import statements. So the manifest the
+entry declarations are never used. The reason the packages are included in the
+isolated output is to instruct package manager to install their dependencies.
+
+## Working with Firebase
+
+> !! There is now
+> [a fork of firebase-tools](https://github.com/0x80/firebase-tools-with-isolate),
+> where isolate-package is integrated. This is preferred because it simplifies
+> the setup and allows the isolation to run only as part of the deploy process,
+> preserving live code updates when running the local Firebase emulators.
 
-If you experience an issue, you can choose to exclude the lockfile from
-deployment by setting `"excludeLockfile": false` in your isolate.config.json
-file, or make the move to PNPM (recommended).
+### A Quick Start
 
-I am not aware of any code in the official Yarn repository for re-generating a
-lockfile, and I am reluctant to work on this feature based on user-land code.
+If you are not confident that your monorepo setup is solid, please check out my
+in-dept example at [mono-ts](https://github.com/0x80/mono-ts) where many
+different aspects are discussed and `isolate-package` is used to demonstrate
+Firebase deployments.
 
-Personally, I do not think Yarn is very relevant anymore in 2023 and I
-personally recommend switching to PNPM.
+This section describes the steps required for Firebase deployment, assuming:
 
-### A Partial Workaround
+- You use a fairly typical monorepo setup
+- Your `firebase.json` config lives in the root of the package that you like to
+  deploy to Firebase, hereafter referred to as the "target package".
 
-If you can not use a lockfile, because you depend on Yarn, a partial workaround
-would be to declare dependencies using exact versions in your package manifest.
-This doesn't prevent your dependencies-dependencies from installing newer
-versions, like a lockfile would, but at least you minimize the risk of things
-breaking.
+If your setup diverges from a traditional one, please continue reading the
+[Prerequisites](#prerequisites) section.
 
-## Different Package Managers
+1. In the target package, install `isolate-package` and `firebase-tools` by
+   running `pnpm add isolate-package firebase-tools -D` or the Yarn / NPM
+   equivalent. I tend to install firebase-tools as a devDependency in every
+   Firebase package, but you could also use a global install if you prefer that.
+2. In the `firebase.json` config set `"source"` to `"./isolate"` and
+   `"predeploy"` to `["turbo build", "isolate"]` or whatever suits your build
+   tool. The important part here is that isolate is being executed after the
+   build stage.
+3. From the target package folder, you should now be able to deploy with
+   `npx firebase deploy`.
 
-Isolate package has been designed to work with all package managers, although
-[PNPM is recommended](https://pnpm.io/feature-comparison), especially for
-monorepo environments.
+I recommend keeping a `firebase.json` file inside each Firebase package (as
+opposed to the monorepo root), because it allows you to deploy from multiple
+independent packages. It makes it easy to deploy 1st gen functions next to 2nd
+gen functions, deploy different node versions, and decrease the built output
+size and dependency lists for each package, improving deployment and cold-start
+times.
 
-The isolation process will infer the package manager name and version from the
-type of lockfile found and the version that the OS reports for the installed
-executable. This information is then used to change some of its behavior. For
-example, the PNPM `pack` process is preferred over the default NPM `pack` if
-PNPM in used, simply because it seems to be much faster.
+### Deploying from multiple packages
 
-The Firebase cloud deploy pipeline will use the package manager that matches
-lockfile that was found in the deployed package.
+You can deploy to Firebase from multiple packages in your monorepo, in which
+case you co-locate your `firebase.json` file with the source code, and not in
+the root of the monorepo. If you do want to keep the firebase config in the
+root, read the instructions for
+[deploying to Firebase from the root](#deploying-to-firebase-from-the-root).
 
-## Using the Firebase Functions Emulator
+In order to deploy to Firebase, the `functions.source` setting in
+`firebase.json` needs to point to the isolated output folder, which would be
+`./isolate` when using the default configuration.
+
+The `predeploy` phase should first build and then isolate the output.
+
+Here's an example using [Turborepo](https://turbo.build/):
+
+```cjson
+// firebase.json
+{
+  "functions": {
+    "source": "./isolate",
+    "predeploy": ["turbo build", "isolate"]
+  }
+}
+```
+
+With this configuration you can then run `npx firebase deploy --only functions`
+from the package.
+
+If you like to deploy to Firebase Functions from multiple packages you will also
+need to configure a unique `codebase` identifier for each of them. For more
+information,
+[read this](https://firebase.google.com/docs/functions/beta/organize-functions).
+
+Make sure your Firebase package adheres to the things mentioned in
+[prerequisites](#prerequisites) and its package manifest contains the field
+`"main"`, or `"module"` if you set `"type": "module"`, so Firebase knows the
+entry point to your source code.
+
+### Deploying from the root
+
+If, for some reason, you choose to keep the `firebase.json` file in the root of
+the monorepo you will have to place a configuration file called
+`isolate.config.json` in the root with the following content:
+
+```cjson
+// isolate.config.json
+{
+  "targetPackagePath": "./packages/your-firebase-package"
+}
+```
+
+The Firebase configuration should then look something like this:
+
+```cjson
+// firebase.json
+{
+  "functions": {
+    "source": "./packages/your-firebase-package/isolate",
+    "predeploy": ["turbo build", "isolate"]
+  }
+}
+```
+
+### Using the Firebase Functions Emulator
 
 The Firebase functions emulator runs on the code that firebase.json `source`
 points to. Unfortunately, this is the same field as is used for declaring the
@@ -531,37 +530,3 @@ deployment process and the `source` property can still point to the original
 code.
 
 I plan to work on this once isolate-package is bit more mature.
-
-## The internal packages strategy
-
-Recently I changed [my example monorepo setup](https://github.com/0x80/mono-ts)
-to include
-[the internal packages strategy](https://turbo.build/blog/you-might-not-need-typescript-project-references),
-(in which the package manifest entries point directly to TS source files, to
-omit the build step), and I was pleased to discover that the approach is
-compatible with `isolate-packages` with only a single change in configuration.
-
-In summary this is how it works:
-
-1. The package to be deployed lists its internal dependencies as usual, but the
-   package manifests of those dependencies point directly to the Typescript
-   source (and types).
-2. You configure the bundler of your target package to include the source code
-   for those internal packages in its output bundle. In the case of TSUP for the
-   [API service in the mono-ts](https://github.com/0x80/mono-ts/blob/main/services/api/tsup.config.ts)
-   that configuration is: `noExternal: ["@mono/common"]`
-3. When `isolate` runs, it does the exact same thing as always. It will detect
-   the internal packages, copies them to the isolate output folder and adjusts
-   any links.
-4. When deploying to Firebase, the cloud pipeline will treat the package
-   manifest as usual, which installs the listed dependencies and any
-   dependencies listed in the linked internal package manifests.
-
-Steps 3 and 4 are no different from a traditional setup.
-
-Note that the manifests for the internal packages will still point to the
-Typescript source files, but since the shared code was embedded in the deployed
-bundle, they will never be referenced via import statements and as a result the
-entry points remain unused. The only reason the packages are included in the
-isolated output is so that the package manager knows what dependencies to
-install.