@browserless.io/browserless 2.0.0-beta-5 → 2.0.0-beta-7

package/LICENSE

@@ -3,4 +3,6 @@
 This work is dual-licensed under the MongoDB Server-Side Public License OR the browserless commercial license.
 You can choose between one of them if you use this work.
 
+If you want to use browserless to build commercial sites, applications, or in a continuous-integration system that's closed-source then you'll need to purchase a commercial license. This allows you to keep your software proprietary whilst still using browserless. [You can purchase a commercial license here](https://www.browserless.io/contact).
+
 `SPDX-License-Identifier: SSPL-1.0 OR Browserless Commercial License`

package/README.md

@@ -8,14 +8,14 @@
 ![Multi CI](https://github.com/browserless/chrome/actions/workflows/docker-multi.yml/badge.svg)
 
 
-> [Looking for v1.x.x of browserless? Check it out here](https://github.com/browserless/chrome/tree/v1).
-> NOTE: Version 1 is the version we currently still have running on browserless' hosted services.
+> [Looking for v2.x.x of browserless? Check it out here](https://github.com/browserless/browserless).
+> NOTE: Version 1 is the version we currently still have running on browserless' hosted services, we would recommend self-hosted users migrate to v2.
 
-browserless is a web-based service that allows for remote clients to connect and execute headless work; all inside of docker. It supports new libraries like Puppeteer and Playwright, aiming to replace antiquated or in-house systems. We also bundle numerous handy REST-based APIs for doing more common actions like data collection, PDF generation and more.
+Browserless allows remote clients to connect and execute headless work, all inside of docker. It supports the standard, unforked Puppeteer and Playwright libraries, as well offering REST-based APIs for common actions like data collection, PDF generation and more.
 
-We also take care of other common issues such as missing system-fonts, missing external libraries, and performance improvements. We even handle edge-cases like downloading files, managing sessions, and have a full documentation site built into the project which includes Open API docs.
+We take care of common issues such as missing system-fonts, missing external libraries, and performance improvements, along with edge-cases like downloading files and managing sessions. For details, check out the documentation site built into the project which includes Open API docs.
 
-If you've been struggling to get a browser up and running docker, or scaling out your headless workloads, then browserless was built for you.
+If you've been struggling to deploy headless browsers without running into issues or bloated resource requirements, then Browserless was built for you. Run the browsers in [our cloud](https://browserless.io/) or your own, [free for non-commercial uses](https://github.com/browserless/browserless#licensing).
 # Table of Contents
 
 1. [Features](#features)
@@ -40,14 +40,14 @@ If you've been struggling to get a browser up and running docker, or scaling out
 - Fonts and emoji's working out-of-the-box.
 - Debug Viewer for actively viewing/debugging running sessions.
 - An interactive puppeteer debugger, so you can see what the headless browser is doing and use its DevTools.
-- Works with most headless libraries.
+- Works with unforked Puppeteer and Playwright.
 - Configurable session timers and health-checks to keep things running smoothly.
 - Error tolerant: if Chrome dies it won't.
 - [Support for running and development on Apple's M1 machines](#building-for-arm64-apple-m1-machines)
 
 # How it works
 
-browserless listens for both incoming websocket requests, generally issued by most libraries, as well as pre-build REST APIs to do common functions (PDF generation, images and so on). When a websocket connects to browserless it starts Chrome and proxies your request into it. Once the session is done then it closes and awaits for more connections. Some libraries use Chrome's HTTP endpoints, like `/json` to inspect debug-able targets, which browserless also supports.
+Browserless listens for both incoming websocket requests, generally issued by most libraries, as well as pre-build REST APIs to do common functions (PDF generation, images and so on). When a websocket connects to Browserless it starts Chrome and proxies your request into it. Once the session is done then it closes and awaits for more connections. Some libraries use Chrome's HTTP endpoints, like `/json` to inspect debug-able targets, which Browserless also supports.
 
 You still execute the script itself which gives you total control over what library you want to choose and when to do upgrades. This also comes with the benefit of keep your code proprietary and able to run on numerous platforms. We simply take care of all the browser-aspects and offer a management layer on top of the browser.
 
@@ -71,7 +71,7 @@ If you're interested in using this image for commercial aspects, then please rea
 
 # Puppeteer
 
-Puppeteer allows you to specify a remote location for chrome via the `browserWSEndpoint` option. Setting this for browserless is a single line of code change.
+Puppeteer allows you to specify a remote location for chrome via the `browserWSEndpoint` option. Setting this for Browserless is a single line of code change.
 
 **Before**
 ```js
@@ -114,13 +114,13 @@ Running Chrome on lambda or on your own is a fantastic idea but in practice is q
 
 Getting Chrome running well in docker is also a challenge as there's quiet a few packages you need in order to get Chrome running. Once that's done then there's still missing fonts, getting libraries to work with it, and having limitations on service reliability. This is also ignoring CVEs, access-controls, and scaling strategies.
 
-All of these issues prompted us to build a first-class image and workflow for interacting with Chrome in a more streamlined way. With browserless you never have to worry about fonts, extra packages, library support, security, or anything else. It just works reliably like any other modern web service. On top of that it comes with a prescribed approach on how you interact with Chrome, which is through socket connections (similar to a database or any other external appliance). What this means is that you get the ability to drive Chrome remotely without having to do updates/releases to the thing that runs Chrome since it's divorced from your application.
+All of these issues prompted us to build a first-class image and workflow for interacting with Chrome in a more streamlined way. With Browserless you never have to worry about fonts, extra packages, library support, security, or anything else. It just works reliably like any other modern web service. On top of that it comes with a prescribed approach on how you interact with Chrome, which is through socket connections (similar to a database or any other external appliance). What this means is that you get the ability to drive Chrome remotely without having to do updates/releases to the thing that runs Chrome since it's divorced from your application.
 
 # Licensing
 
 SPDX-License-Identifier: SSPL-1.0 OR Browserless Commercial License.
 
-If you want to use browserless to build commercial sites, applications, or in a continuous-integration system that's closed-source then you'll need to purchase a commercial license. This allows you to keep your software proprietary whilst still using browserless. [You can purchase a commercial license here](https://www.browserless.io/contact). A commercial license grants you:
+If you want to use Browserless to build commercial sites, applications, or in a continuous-integration system that's closed-source then you'll need to purchase a commercial license. This allows you to keep your software proprietary whilst still using browserless. [You can purchase a commercial license here](https://www.browserless.io/contact). A commercial license grants you:
 
 - Priority support on issues and features.
 - On-premise running as well as running on public cloud providers for commercial/CI purposes for proprietary systems.
@@ -129,4 +129,4 @@ If you want to use browserless to build commercial sites, applications, or in a
 
 Not only does it grant you a license to run such a critical piece of infrastructure, but you are also supporting further innovation in this space and our ability to contribute to it.
 
-If you are creating an open source application under a license compatible with the Server Side License 1.0, you may use browserless under those terms.
+If you are creating an open source application under a license compatible with the Server Side License 1.0, you may use Browserless under those terms.

package/bin/browserless.js

@@ -1,12 +1,16 @@
 #!/usr/bin/env node
 /* eslint-disable no-undef */
 'use strict';
+process.env.DEBUG = process.env.DEBUG || 'browserless*';
+
+import { readFile, writeFile } from 'fs/promises';
 import { Browserless } from '@browserless.io/browserless';
 import buildOpenAPI from '../scripts/build-open-api.js';
 import buildSchemas from '../scripts/build-schemas.js';
 
 import { createInterface } from 'readline';
 import debug from 'debug';
+import { dedent } from '../build/utils.js';
 import { fileURLToPath } from 'url';
 import fs from 'fs/promises';
 import path from 'path';
@@ -17,9 +21,13 @@ const promptLog = debug('browserless:prompt');
 
 const __dirname = path.dirname(fileURLToPath(import.meta.url));
 const cmd = process.argv[2];
+const subCMD = process.argv[3];
 const cwd = process.cwd();
-const allowedCMDs = ['build', 'dev', 'docker', 'start'];
+const allowedCMDs = ['build', 'dev', 'docker', 'start', 'create', 'help'];
 const srcDir = path.join(cwd, 'build');
+const packageJSON = readFile(path.join(__dirname, '..', 'package.json')).then(
+  (r) => JSON.parse(r.toString()),
+);
 
 if (!allowedCMDs.includes(cmd)) {
   throw new Error(
@@ -60,6 +68,22 @@ const importClassOverride = async (files, className) => {
   return (await import(classModuleFullFilePath)).default;
 };
 
+const installDependencies = async (workingDirectory) =>
+  new Promise((resolve, reject) => {
+    spawn('npm', ['i'], {
+      cwd: workingDirectory,
+      stdio: 'inherit',
+    }).once('close', (code) => {
+      if (code === 0) {
+        log(`Successfully installed Dependencies.`);
+        return resolve();
+      }
+      return reject(
+        `Error when installing dependencies, see output for more details`,
+      );
+    });
+  });
+
 const buildDockerImage = async (cmd) => {
   new Promise((resolve, reject) => {
     const [docker, ...args] = cmd.split(' ');
@@ -94,7 +118,7 @@ const buildTypeScript = async () =>
   });
 
 const getSourceFiles = async () => {
-  const files = await fs.readdir(srcDir);
+  const files = await fs.readdir(srcDir, { recursive: true });
   const [httpRoutes, webSocketRoutes] = files.reduce(
     ([httpRoutes, webSocketRoutes], file) => {
       const parsed = path.parse(file);
@@ -210,25 +234,25 @@ const start = async (dev = false) => {
 
   log(`Starting Browserless`);
 
-  const config = isConstructor(Config) ? new Config() : config;
-  const metrics = isConstructor(Metrics) ? new Metrics() : metrics;
-  const token = isConstructor(Token) ? new Token(config) : token;
-  const webhooks = isConstructor(Webhooks) ? new Webhooks(config) : webhooks;
+  const config = isConstructor(Config) ? new Config() : Config;
+  const metrics = isConstructor(Metrics) ? new Metrics() : Metrics;
+  const token = isConstructor(Token) ? new Token(config) : Token;
+  const webhooks = isConstructor(Webhooks) ? new Webhooks(config) : Webhooks;
   const browserManager = isConstructor(BrowserManager)
     ? new BrowserManager(config)
-    : browserManager;
+    : BrowserManager;
   const monitoring = isConstructor(Monitoring)
     ? new Monitoring(config)
-    : monitoring;
+    : Monitoring;
   const fileSystem = isConstructor(FileSystem)
     ? new FileSystem(config)
-    : fileSystem;
+    : FileSystem;
   const limiter = isConstructor(Limiter)
     ? new Limiter(config, metrics, monitoring, webhooks)
-    : limiter;
+    : Limiter;
   const router = isConstructor(Router)
     ? new Router(config, browserManager, limiter)
-    : router;
+    : Router;
 
   const browserless = new Browserless({
     browserManager,
@@ -350,6 +374,132 @@ const buildDocker = async () => {
   }
 };
 
+const create = async () => {
+  const validNameRegex = /^[a-zA-Z0-9-_]+$/gi;
+  const directory = (
+    await prompt('What should we name this project (hyphens are ok)?')
+  ).trim();
+  const scaffoldLocation = path.join(__dirname, 'scaffold');
+
+  if (!directory) {
+    throw new Error(`A valid name is required.`);
+  }
+
+  const isValidDirectory = validNameRegex.test(directory);
+
+  if (!isValidDirectory) {
+    throw new Error(`Name must not include special characters.`);
+  }
+
+  const installPath = path.join(cwd, directory);
+  log(`Creating folder "${installPath}"...`);
+  await fs.mkdir(installPath);
+
+  log(`Copying Project Dependencies...`);
+  const sdkFiles = await fs.readdir(scaffoldLocation, { recursive: true });
+  for (const sdkFile of sdkFiles) {
+    const from = path.join(scaffoldLocation, sdkFile);
+    const to = path.join(installPath, sdkFile);
+    if (sdkFile === 'package.json') {
+      const sdkPackageJSONTemplate = (await readFile(from)).toString();
+      const { version } = await packageJSON;
+      const sdkPackageJSON = sdkPackageJSONTemplate.replace(
+        '${BROWSERLESS_VERSION}',
+        version,
+      );
+      await writeFile(to, sdkPackageJSON);
+    } else if ((await fs.lstat(from)).isDirectory()) {
+      await fs.mkdir(to);
+    } else {
+      await fs.copyFile(from, to);
+    }
+  }
+
+  log(`Installing npm modules...`);
+  await installDependencies(installPath);
+
+  log(
+    `Done! You can now open "${installPath}" in an editor of your choice. Make sure to check out the README and update the package.json file!`,
+  );
+};
+
+const help = () => {
+  if (subCMD) {
+    if (!allowedCMDs.includes(subCMD)) {
+      throw new Error(`Unknown command of "${subCMD}" passed.`);
+    }
+
+    switch (subCMD) {
+      case 'start':
+        console.log(dedent`
+        Usage: npx @browserless.io/browserless start
+
+        Description: Starts the HTTP server without building source.
+          Useful for restarting a prior build, testing quickly, or
+          running without packaging into a docker image.
+      `);
+        break;
+
+      case 'dev':
+        console.log(dedent`
+        Usage: npx @browserless.io/browserless dev
+
+        Description: Builds the TypeScript files, compiles runtime
+          route validation, generates the OpenAPI JSON document,
+          and starts the development server at localhost:3000.
+      `);
+        break;
+
+      case 'build':
+        console.log(dedent`
+        Usage: npx @browserless.io/browserless build
+
+        Description: Builds the TypeScript files, compiles runtime
+          route validation, generates the OpenAPI JSON document,
+          and exits. Useful for testing full compilation.
+      `);
+        break;
+
+      case 'docker':
+        console.log(dedent`
+        Usage: npx @browserless.io/browserless docker
+
+        Description: Builds a docker image from source. This command is hybrid
+          in that it can be either interactive or use the switches listed below.
+
+        Options:
+          --from        The Browserless docker image to extend from (ghcr.io/browserless/multi:latest).
+          --action      One of "push" or "load" to load or push to a registry.
+          --tag         The full tag, including version, to name the image (IE: my-bless/chrome:latest).
+          --platform    A comma-separated list of platforms to build for.
+          --proceed     Proceed with building the image without prompting.
+      `);
+        break;
+
+      case 'create':
+        console.log(dedent`
+        Usage: npx @browserless.io/browserless create
+
+        Description: Creates a new project with interactive prompts.
+      `);
+        break;
+    }
+
+    return;
+  }
+
+  console.log(dedent`
+    Usage: npx @browserless.io/browserless [command] [arguments]
+
+    Options:
+      create    Creates a new scaffold project, installs dependencies, and exits.
+      dev       Compiles TypeScript, generates build assets and starts the server.
+      build     Compiles TypeScript, generates build assets and exits.
+      docker    Generates a docker image.
+      start     Starts the http server with already-built assets.
+  `);
+};
+
 switch (cmd) {
   case 'start':
     start(false);
@@ -366,4 +516,12 @@ switch (cmd) {
   case 'docker':
     buildDocker();
     break;
+
+  case 'create':
+    create();
+    break;
+
+  default:
+    help();
+    break;
 }