@browserless.io/browserless 2.0.0-beta-7 → 2.1.0

package/README.md

@@ -8,8 +8,7 @@
 ![Multi CI](https://github.com/browserless/chrome/actions/workflows/docker-multi.yml/badge.svg)
 
 
-> [Looking for v2.x.x of browserless? Check it out here](https://github.com/browserless/browserless).
-> NOTE: Version 1 is the version we currently still have running on browserless' hosted services, we would recommend self-hosted users migrate to v2.
+> [Looking for v1.x.x of browserless? You can find it here](https://github.com/browserless/chrome/tree/v1), although we recommend migrating to v2.
 
 Browserless allows remote clients to connect and execute headless work, all inside of docker. It supports the standard, unforked Puppeteer and Playwright libraries, as well offering REST-based APIs for common actions like data collection, PDF generation and more.
 
@@ -24,14 +23,15 @@ If you've been struggling to deploy headless browsers without running into issue
 4. [Hosting](#hosting-providers)
 5. [Puppeteer](#puppeteer)
 6. [Playwright](#playwright)
-7. [Licensing](#licensing)
-8. [Changelog](https://github.com/browserless/chrome/blob/master/CHANGELOG.md)
+7. [Extending with NodeJS SDK](#extending-nodejs-skd)
+8. [Licensing](#licensing)
+9. [Changelog](https://github.com/browserless/chrome/blob/master/CHANGELOG.md)
 
 ## External links
 
 1. [Full documentation site](https://www.browserless.io/docs/start)
 2. [Live Debugger (using browserless.io)](https://chrome.browserless.io/)
-3. [Docker](https://github.com/browserless/chrome/pkgs/container/basic)
+3. [Docker](https://github.com/browserless/browserless/pkgs/container/base)
 4. [Slack](https://join.slack.com/t/browserless/shared_invite/enQtMzA3OTMwNjA3MzY1LTRmMWU5NjQ0MTQ2YTE2YmU3MzdjNmVlMmU4MThjM2UxODNmNzNlZjVkY2U2NjdkMzYyNTgyZTBiMmE3Nzg0MzY)
 
 # Features
@@ -57,7 +57,7 @@ You still execute the script itself which gives you total control over what libr
 
 1. `docker run -p 3000:3000 ghcr.io/browserless/chrome`
 2. Visit `http://localhost:3000/docs` to see the documentation site.
-3. See more at our [docker package](https://github.com/browserless/chrome/pkgs/container/basic).
+3. See more at our [docker package](https://github.com/browserless/browserless/pkgs/container/base).
 
 # Hosting Providers
 
@@ -102,6 +102,10 @@ const browser = await pw.chromium.connectOverCDP('ws://localhost:3000');
 
 After that, the rest of your code remains the same with no other changes required.
 
+# Extending (NodeJS SKD)
+
+Browserless comes with built-in extension capabilities, and allows for extending nearly any aspect of the system (for Version 2+). For more details on how to write your own routes, build docker images, and more, [see our SDK README.md](/bin/scaffold/README.md) or simply run "npx @browserless.io/browserless create" in a terminal and follow the onscreen prompts.
+
 # Usage with other libraries
 
 Most libraries allow you to specify a remote instance of Chrome to interact with. They are either looking for a websocket endpoint, a host and port, or some address. Browserless supports these by default, however if you're having issues please make an issue in this project and we'll try and work with the library authors to get them integrated with browserless. Please note that in V2 we no longer support selenium or webdriver integrations.

package/bin/browserless.js

@@ -1,8 +1,6 @@
 #!/usr/bin/env node
 /* eslint-disable no-undef */
 'use strict';
-process.env.DEBUG = process.env.DEBUG || 'browserless*';
-
 import { readFile, writeFile } from 'fs/promises';
 import { Browserless } from '@browserless.io/browserless';
 import buildOpenAPI from '../scripts/build-open-api.js';
@@ -16,8 +14,10 @@ import fs from 'fs/promises';
 import path from 'path';
 import { spawn } from 'child_process';
 
-const log = debug('browserless:sdk:log');
-const promptLog = debug('browserless:prompt');
+debug.enable('browserless*');
+
+const log = debug('browserless.io:sdk:log');
+const promptLog = debug('browserless.io:prompt');
 
 const __dirname = path.dirname(fileURLToPath(import.meta.url));
 const cmd = process.argv[2];

package/bin/scaffold/README.md

@@ -106,16 +106,16 @@ By default most commands are non-interactive, such as the `build` and `dev` comm
 
 ## Routing
 
-Routing is, simply, a plain-old JavaScript object with certain expected properties. Many of the features of Browserless are exposed as options on routes so you can define many types of functionality with just a route definition.
+Routing is based upon the JavaScript `class` fundamentals, and extends core classes inside of Browserless. Many of the features of Browserless are exposed as options on routes so you can define many types of functionality with just a simple route definition.
 
-Browserless has 4 different types of routes:
+Browserless has 4 different types of primitive routes:
 
-- HTTP Routes that don't need a browser to run.
-- HTTP Routes that do need a browser to run.
-- WebSocket routes that don't a browser.
-- WebSockets that need a browser.
+- HTTP Routes.
+- HTTP Routes that require a browser.
+- WebSockets Routes.
+- WebSocket Routes require a browser.
 
-We use this same semantic in our own codebase, so feel free to see how those work in our open-source projects. All routes are TypeScript and all our modules are documented, so you should be able to effectively write routes and modules with your code editor and not necessarily need these examples open. Below are a few examples:
+Internally, we use this same class-based system, so feel free to see how those work in our open-source repositories. All routes are TypeScript-based and all our modules are documented, so you should be able to effectively write routes and modules with your code editor and not necessarily need these examples open. Below are a few examples:
 
 ### Basic HTTP Route
 ```ts

package/bin/scaffold/tsconfig.json

@@ -1,4 +1,4 @@
 {
   "extends": "./node_modules/@browserless.io/browserless/tsconfig.json",
-  "include": ["./src/**/*"]
+  "include": ["./src/**/*"],
 }

package/browser.json

@@ -1,7 +1,7 @@
 {
-  "Browser": "HeadlessChrome/120.0.6099.28",
+  "Browser": "HeadlessChrome/121.0.6167.57",
   "Protocol-Version": "1.3",
-  "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36",
-  "WebKit-Version": "537.36 (@f4095e9665f7d7a2531edefcea119d45d899d95b)",
+  "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/121.0.6167.57 Safari/537.36",
+  "WebKit-Version": "537.36 (@add6d6ffbc3a1c7e78cc15e6ba2dcb15208bedd5)",
   "webSocketDebuggerUrl": "ws://localhost:3000"
 }

package/build/browsers/cdp-chromium.js

@@ -229,11 +229,10 @@ export class CDPChromium extends EventEmitter {
         if (!this.browserWSEndpoint) {
             return null;
         }
+        const serverURL = new URL(this.config.getExternalWebSocketAddress());
         const wsURL = new URL(this.browserWSEndpoint);
-        const serverURL = new URL(this.config.getExternalAddress());
         wsURL.hostname = serverURL.hostname;
         wsURL.port = serverURL.port;
-        wsURL.protocol = serverURL.protocol === 'https' ? 'wss' : 'ws';
         if (token) {
             wsURL.searchParams.set('token', token);
         }

package/build/browsers/index.d.ts

@@ -18,6 +18,15 @@ export declare class BrowserManager {
      * @returns Promise<string> of the fully-qualified path of the directory
      */
     protected generateDataDir: (sessionId?: string) => Promise<string>;
+    getVersionJSON: () => Promise<{
+        Browser: string;
+        'Debugger-Version': string;
+        'Protocol-Version': string;
+        'User-Agent': string;
+        'V8-Version': string;
+        'WebKit-Version': string;
+        webSocketDebuggerUrl: string;
+    }>;
     private generateSessionJson;
     close: (browser: BrowserInstance, session: BrowserlessSession) => Promise<void>;
     getAllSessions: () => Promise<BrowserlessSessionJSON[]>;

package/build/browsers/index.js

@@ -41,6 +41,30 @@ export class BrowserManager {
         });
         return dataDirPath;
     };
+    getVersionJSON = async () => {
+        const browser = new CDPChromium({
+            blockAds: false,
+            config: this.config,
+            record: false,
+            userDataDir: null,
+        });
+        await browser.launch();
+        const wsEndpoint = browser.wsEndpoint();
+        if (!wsEndpoint) {
+            throw new Error('There was an error launching the browser');
+        }
+        const { port } = new URL(wsEndpoint);
+        const res = await fetch(`http://127.0.0.1:${port}/json/version`);
+        const meta = await res.json();
+        browser.close();
+        const { 'WebKit-Version': webkitVersion } = meta;
+        const debuggerVersion = webkitVersion.match(/\s\(@(\b[0-9a-f]{5,40}\b)/)[1];
+        return {
+            ...meta,
+            'Debugger-Version': debuggerVersion,
+            webSocketDebuggerUrl: this.config.getExternalWebSocketAddress(),
+        };
+    };
     generateSessionJson = async (browser, session) => {
         const serverAddress = this.config.getExternalAddress();
         const sessions = [

package/build/browsers/playwright-chromium.js

@@ -72,11 +72,10 @@ export class PlaywrightChromium extends EventEmitter {
         if (!this.browserWSEndpoint) {
             return null;
         }
+        const serverURL = new URL(this.config.getExternalWebSocketAddress());
         const wsURL = new URL(this.browserWSEndpoint);
-        const serverURL = new URL(this.config.getExternalAddress());
         wsURL.hostname = serverURL.hostname;
         wsURL.port = serverURL.port;
-        wsURL.protocol = serverURL.protocol === 'https' ? 'wss' : 'ws';
         if (token) {
             wsURL.searchParams.set('token', token);
         }

package/build/browsers/playwright-firefox.js

@@ -67,11 +67,10 @@ export class PlaywrightFirefox extends EventEmitter {
         if (!this.browserWSEndpoint) {
             return null;
         }
+        const serverURL = new URL(this.config.getExternalWebSocketAddress());
         const wsURL = new URL(this.browserWSEndpoint);
-        const serverURL = new URL(this.config.getExternalAddress());
         wsURL.hostname = serverURL.hostname;
         wsURL.port = serverURL.port;
-        wsURL.protocol = serverURL.protocol === 'https' ? 'wss' : 'ws';
         if (token) {
             wsURL.searchParams.set('token', token);
         }

package/build/browsers/playwright-webkit.js

@@ -67,11 +67,10 @@ export class PlaywrightWebkit extends EventEmitter {
         if (!this.browserWSEndpoint) {
             return null;
         }
+        const serverURL = new URL(this.config.getExternalWebSocketAddress());
         const wsURL = new URL(this.browserWSEndpoint);
-        const serverURL = new URL(this.config.getExternalAddress());
         wsURL.hostname = serverURL.hostname;
         wsURL.port = serverURL.port;
-        wsURL.protocol = serverURL.protocol === 'https' ? 'wss' : 'ws';
         if (token) {
             wsURL.searchParams.set('token', token);
         }

package/build/config.d.ts

@@ -124,6 +124,15 @@ export declare class Config extends EventEmitter {
      * @returns {string} The URL to reach the server
      */
     getExternalAddress: () => string;
+    /**
+     * Returns the the fully-qualified WebSocket URL for the
+     * external address that browserless might be
+     * running behind *or* the server address if
+     * no external URL is provided.
+     *
+     * @returns {string} The URL to reach the server
+     */
+    getExternalWebSocketAddress: () => string;
     /**
      * When CORS is enabled, returns relevant CORS headers
      * to requests and for the OPTIONS call. Values can be

package/build/config.js

@@ -335,6 +335,21 @@ export class Config extends EventEmitter {
      * @returns {string} The URL to reach the server
      */
     getExternalAddress = () => this.external ?? this.getServerAddress();
+    /**
+     * Returns the the fully-qualified WebSocket URL for the
+     * external address that browserless might be
+     * running behind *or* the server address if
+     * no external URL is provided.
+     *
+     * @returns {string} The URL to reach the server
+     */
+    getExternalWebSocketAddress = () => {
+        const httpAddress = new URL(this.external ?? this.getServerAddress());
+        httpAddress.protocol = httpAddress.protocol.startsWith('https')
+            ? 'wss:'
+            : 'ws:';
+        return httpAddress.href;
+    };
     /**
      * When CORS is enabled, returns relevant CORS headers
      * to requests and for the OPTIONS call. Values can be