@procedures/browser 0.55.2 → 0.56.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,25 @@
1
1
  # @procedures/browser
2
2
 
3
+ ## 0.56.0
4
+
5
+ ### Patch Changes
6
+
7
+ - 7498bc1: `@procedures/http` has a local forward proxy that applies the host policy to each request of another program. `@procedures/browser` uses it for the new option `hostPolicy` of each render. The change adds names and is backward compatible, so it is a patch.
8
+
9
+ - `startPolicyProxy({ policy, onRefusal })` starts the proxy on `127.0.0.1`, on a free port. It gives `host`, `port`, `url`, `refusals`, `requests` and `close()`.
10
+ - The proxy takes `CONNECT` tunnels and plain requests with an absolute `http:` URL. It examines each host with `checkHost`, and connects only to the checked addresses.
11
+ - A refused host gets the status 403. `refusals` keeps the first `MAX_PROXY_REFUSALS` refusals, and `onRefusal` gets each refusal.
12
+ - The options `hostPolicy` and `onHostRefusal` of each render start the proxy for the run, and stop it at the end, also after an error.
13
+ - `policyProxyArgs(proxyUrl)` gives the flags of the browser: the proxy, the bypass rule `<-loopback>`, no QUIC, no UDP for WebRTC and no DNS look-up.
14
+ - With `hostPolicy`, a flag of `args` that can undo these flags gives an error before the browser starts.
15
+ - A Chrome or Edge policy of the computer (`ProxySettings` or `ProxyMode`) replaces `--proxy-server` with no error. Thus a render of an `http:` or `https:` URL fails when the proxy gets no request.
16
+ - `@procedures/browser` now depends on `@procedures/http`.
17
+ - 13996cf: An automated dependency update changes the shipped files, so this PATCH ships them.
18
+ - Updated dependencies [7498bc1]
19
+ - Updated dependencies [13996cf]
20
+ - @procedures/http@0.56.0
21
+ - @procedures/shell@0.56.0
22
+
3
23
  ## 0.55.2
4
24
 
5
25
  ### Patch Changes
package/README.md CHANGED
@@ -65,6 +65,8 @@ const { path, bytes } = await renderPdf({ html: "<h1>Menu</h1><p>Soup of the day
65
65
  | `timeout` | The time limit, in milliseconds. The default is 60000. |
66
66
  | `signal` | When the signal aborts, the browser stops. |
67
67
  | `runner` | The function that runs the browser. The default is `runCommand` of `@procedures/shell`. The tests give their own. |
68
+ | `hostPolicy` | The host policy of each request of the browser, from `@procedures/http`. The section [The host policy](#the-host-policy) tells more. |
69
+ | `onHostRefusal` | This function gets each request that the host policy refused. It needs `hostPolicy`. |
68
70
 
69
71
  - The browser starts through `runCommand` of `@procedures/shell`, not through a shell. At the time limit, or when the signal aborts, `runCommand` stops the process tree of the browser.
70
72
  - The temporary profile keeps the render apart from a browser that the user has open.
@@ -72,6 +74,46 @@ const { path, bytes } = await renderPdf({ html: "<h1>Menu</h1><p>Soup of the day
72
74
  - An existing output file changes only after the check of the new file. Thus a failed render keeps an old output file.
73
75
  - Each render removes its temporary page and profile, also after an error. For a file that Windows holds, the removal tries again 3 times.
74
76
 
77
+ ## The host policy
78
+
79
+ A check of the address before the render is not enough. A redirect, a resource of the page or a second DNS answer can reach a local address. The option `hostPolicy` applies the policy to each request of the browser.
80
+
81
+ ```ts
82
+ import { createHostPolicy } from "@procedures/http";
83
+ import { renderScreenshotBytes } from "@procedures/browser";
84
+
85
+ const { data } = await renderScreenshotBytes({
86
+ url: "https://news.example/",
87
+ format: "jpeg",
88
+ hostPolicy: createHostPolicy({ allow: ["cdn.intranet.example"] }),
89
+ onHostRefusal: (refusal) => console.warn(refusal.method, refusal.target, refusal.reason),
90
+ });
91
+ ```
92
+
93
+ - The run starts the policy proxy of `@procedures/http` (`startPolicyProxy`) before the browser. The proxy stops at the end of the run, also after an error.
94
+ - The proxy gives the status 403 to each host that the policy refuses. The browser shows an error for that request.
95
+ - `DEFAULT_HOST_POLICY` of `@procedures/http` lets the browser reach each public host, and no other host.
96
+ - The option applies to each render: the screenshot and the PDF file.
97
+ - A Chrome or Edge policy of the computer (`ProxySettings` or `ProxyMode`) replaces the flag `--proxy-server`, and the browser gives no error. Thus the run counts the requests of the proxy.
98
+ - When the page is an `http:` or `https:` URL and the proxy gets no request, the render fails. The page of an `html` or `file` source can have no request, so this check does not apply to it.
99
+
100
+ `policyProxyArgs(proxyUrl)` gives the flags of the browser. They come before the flags of `args`:
101
+
102
+ | Flag | What it does |
103
+ |---|---|
104
+ | `--proxy-server=<url of the proxy>` | The browser sends each `http:`, `https:`, `ws:` and `wss:` request to the proxy. The proxy resolves the names, not the browser. |
105
+ | `--proxy-bypass-list=<-loopback>` | The browser also sends the requests for `localhost`, the loopback addresses and the link-local addresses to the proxy. |
106
+ | `--disable-quic` | The browser does not use QUIC, so no request goes over UDP around the proxy. |
107
+ | `--force-webrtc-ip-handling-policy=disable_non_proxied_udp` | WebRTC does not use UDP, and does not show a local address. |
108
+ | `--host-resolver-rules=MAP * ~NOTFOUND , EXCLUDE 127.0.0.1` | The browser does no DNS look-up. The only exception is the address of the proxy. |
109
+
110
+ - With `hostPolicy`, `args` cannot have a flag that can undo these flags. The run gives an error before it starts the browser.
111
+ - These flags are `--proxy-server`, `--proxy-bypass-list`, `--no-proxy-server`, `--proxy-pac-url`, `--proxy-auto-detect`, `--use-system-proxy-resolver` and `--winhttp-proxy-resolver`.
112
+ - The other flags are `--enable-quic`, `--origin-to-force-quic-on`, `--force-webrtc-ip-handling-policy`, `--host-resolver-rules` and `--host-rules`.
113
+ - The check finds a flag with the prefix `--`, `-` or `/`, and ignores the letter case.
114
+ - WebTransport fails through a proxy, so a page cannot use it.
115
+ - The sources of the flags are the Chromium documents `net/docs/proxy.md`, `chrome_switches.h`, `network_switch_list.h`, `network_switches.cc`, `content_switches.cc` and `webrtc_ip_handling_policy.h`.
116
+
75
117
  ## The browser
76
118
 
77
119
  `findBrowser(options)` gives the path of the browser program, or `undefined`. The search uses the first of these:
package/dist/index.d.ts CHANGED
@@ -1,5 +1,5 @@
1
1
  export * from "./browser.js";
2
- export { DEFAULT_TIMEOUT_MS, type CommandRunner, type HeadlessOptions, type RenderResult } from "./run.js";
2
+ export { DEFAULT_TIMEOUT_MS, policyProxyArgs, type CommandRunner, type HeadlessOptions, type RenderResult } from "./run.js";
3
3
  export { isPdfFile, pdfArgs, renderPdf, type RenderPdfOptions, type RenderPdfResult } from "./pdf.js";
4
4
  export * from "./screenshot.js";
5
5
  export * from "./bookmarks.js";
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AACA,cAAc,cAAc,CAAC;AAC7B,OAAO,EAAE,kBAAkB,EAAE,KAAK,aAAa,EAAE,KAAK,eAAe,EAAE,KAAK,YAAY,EAAE,MAAM,UAAU,CAAC;AAC3G,OAAO,EAAE,SAAS,EAAE,OAAO,EAAE,SAAS,EAAE,KAAK,gBAAgB,EAAE,KAAK,eAAe,EAAE,MAAM,UAAU,CAAC;AACtG,cAAc,iBAAiB,CAAC;AAEhC,cAAc,gBAAgB,CAAC;AAC/B,cAAc,eAAe,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AACA,cAAc,cAAc,CAAC;AAC7B,OAAO,EAAE,kBAAkB,EAAE,eAAe,EAAE,KAAK,aAAa,EAAE,KAAK,eAAe,EAAE,KAAK,YAAY,EAAE,MAAM,UAAU,CAAC;AAC5H,OAAO,EAAE,SAAS,EAAE,OAAO,EAAE,SAAS,EAAE,KAAK,gBAAgB,EAAE,KAAK,eAAe,EAAE,MAAM,UAAU,CAAC;AACtG,cAAc,iBAAiB,CAAC;AAEhC,cAAc,gBAAgB,CAAC;AAC/B,cAAc,eAAe,CAAC"}
package/dist/index.js CHANGED
@@ -1,6 +1,6 @@
1
1
  // The package has no procedures. It gives functions only: the search, the run and the renders.
2
2
  export * from "./browser.js";
3
- export { DEFAULT_TIMEOUT_MS } from "./run.js";
3
+ export { DEFAULT_TIMEOUT_MS, policyProxyArgs } from "./run.js";
4
4
  export { isPdfFile, pdfArgs, renderPdf } from "./pdf.js";
5
5
  export * from "./screenshot.js";
6
6
  // The bookmarks files of Chrome, Edge and Chromium: the parse (also on the browser entry ./bookmarks) and the search on the disk
package/dist/index.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,+FAA+F;AAC/F,cAAc,cAAc,CAAC;AAC7B,OAAO,EAAE,kBAAkB,EAA+D,MAAM,UAAU,CAAC;AAC3G,OAAO,EAAE,SAAS,EAAE,OAAO,EAAE,SAAS,EAA+C,MAAM,UAAU,CAAC;AACtG,cAAc,iBAAiB,CAAC;AAChC,iIAAiI;AACjI,cAAc,gBAAgB,CAAC;AAC/B,cAAc,eAAe,CAAC"}
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,+FAA+F;AAC/F,cAAc,cAAc,CAAC;AAC7B,OAAO,EAAE,kBAAkB,EAAE,eAAe,EAA+D,MAAM,UAAU,CAAC;AAC5H,OAAO,EAAE,SAAS,EAAE,OAAO,EAAE,SAAS,EAA+C,MAAM,UAAU,CAAC;AACtG,cAAc,iBAAiB,CAAC;AAChC,iIAAiI;AACjI,cAAc,gBAAgB,CAAC;AAC/B,cAAc,eAAe,CAAC"}
package/dist/run.d.ts CHANGED
@@ -5,7 +5,12 @@
5
5
  *
6
6
  * The browser starts through `runCommand` of `@procedures/shell`, not through a shell. A time limit
7
7
  * and the signal of the call stop its process tree.
8
+ *
9
+ * With the option `hostPolicy`, the run starts a policy proxy of `@procedures/http`. The browser
10
+ * sends each request through the proxy, and the proxy applies the host policy to each request.
11
+ * When the page is an `http:` or `https:` URL and the proxy gets no request, the run fails.
8
12
  */
13
+ import { type HostPolicy, type ProxyRefusal } from "@procedures/http";
9
14
  import { type CommandOptions, type CommandResult } from "@procedures/shell/command";
10
15
  import { type FindBrowserOptions } from "./browser.js";
11
16
  /** The default time limit of one render, in milliseconds. */
@@ -24,7 +29,30 @@ export interface HeadlessOptions extends FindBrowserOptions {
24
29
  readonly signal?: AbortSignal | undefined;
25
30
  /** The function that runs the browser (default: `runCommand`). The tests give their own. */
26
31
  readonly runner?: CommandRunner | undefined;
32
+ /**
33
+ * The host policy of each request of the browser. With it, each request goes through a policy
34
+ * proxy, also each redirect and each resource of the page. The proxy gives the status 403 to
35
+ * each host that the policy refuses.
36
+ */
37
+ readonly hostPolicy?: HostPolicy | undefined;
38
+ /** This function gets each request that the host policy refused. It needs `hostPolicy`. */
39
+ readonly onHostRefusal?: ((refusal: ProxyRefusal) => void) | undefined;
27
40
  }
41
+ /**
42
+ * This function gives the flags that send each request of the browser through the proxy at
43
+ * `proxyUrl`, and that close the paths around the proxy.
44
+ *
45
+ * - `--proxy-server` sends each `http:`, `https:`, `ws:` and `wss:` request to the proxy. The proxy
46
+ * resolves each name, not the browser.
47
+ * - `--proxy-bypass-list=<-loopback>` removes the default bypass of `localhost`, of the loopback
48
+ * addresses and of the link-local addresses. Thus these requests also go to the proxy.
49
+ * - `--disable-quic` stops the QUIC protocol, so no request goes over UDP around the proxy.
50
+ * - `--force-webrtc-ip-handling-policy=disable_non_proxied_udp` stops the UDP of WebRTC, and shows no
51
+ * local address.
52
+ * - `--host-resolver-rules` stops each DNS look-up of the browser. The only exception is the address
53
+ * of the proxy.
54
+ */
55
+ export declare function policyProxyArgs(proxyUrl: string): string[];
28
56
  /** The result of a render. */
29
57
  export interface RenderResult {
30
58
  /** The absolute path of the output file. */
@@ -71,6 +99,11 @@ export declare function withTemporaryFolder<T>(prefix: string, folder: string |
71
99
  * An existing output file changes only after the check of the new file. Thus a failed render keeps
72
100
  * an old output file. Without the option `output`, the file goes in a new temporary folder. This
73
101
  * folder stays after a success, and goes after a failure.
102
+ *
103
+ * With the option `hostPolicy`, the browser gets the flags of `policyProxyArgs` before the flags of
104
+ * `args`. The proxy stops at the end of the run, also after an error. When the page is an `http:`
105
+ * or `https:` URL and the proxy gets no request, the render fails. Then the requests of the page
106
+ * did not go through the policy.
74
107
  */
75
108
  export declare function headlessRender(render: HeadlessRender, source: PageSource, options: HeadlessOptions): Promise<RenderResult>;
76
109
  //# sourceMappingURL=run.d.ts.map
package/dist/run.d.ts.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"run.d.ts","sourceRoot":"","sources":["../src/run.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAMH,OAAO,EAAc,KAAK,cAAc,EAAE,KAAK,aAAa,EAAE,MAAM,2BAA2B,CAAC;AAChG,OAAO,EAA4B,KAAK,kBAAkB,EAAE,MAAM,cAAc,CAAC;AAEjF,6DAA6D;AAC7D,eAAO,MAAM,kBAAkB,QAAS,CAAC;AAEzC,4FAA4F;AAC5F,MAAM,MAAM,aAAa,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,cAAc,KAAK,OAAO,CAAC,aAAa,CAAC,CAAC;AAEjG,kCAAkC;AAClC,MAAM,WAAW,eAAgB,SAAQ,kBAAkB;IACzD,oFAAoF;IACpF,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACrC,2DAA2D;IAC3D,QAAQ,CAAC,IAAI,CAAC,EAAE,SAAS,MAAM,EAAE,GAAG,SAAS,CAAC;IAC9C,wDAAwD;IACxD,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACtC,iDAAiD;IACjD,QAAQ,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAC;IAC1C,4FAA4F;IAC5F,QAAQ,CAAC,MAAM,CAAC,EAAE,aAAa,GAAG,SAAS,CAAC;CAC7C;AAED,8BAA8B;AAC9B,MAAM,WAAW,YAAY;IAC3B,4CAA4C;IAC5C,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,6CAA6C;IAC7C,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,8CAA8C;IAC9C,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;CAC1B;AAED,kFAAkF;AAClF,MAAM,MAAM,UAAU,GAAG;IAAE,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAA;CAAE,GAAG;IAAE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAA;CAAE,GAAG;IAAE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAA;CAAE,CAAC;AAE1G,qFAAqF;AACrF,MAAM,WAAW,cAAc;IAC7B,oFAAoF;IACpF,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,oGAAoG;IACpG,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,gEAAgE;IAChE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,8FAA8F;IAC9F,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,SAAS,MAAM,EAAE,GAAG,MAAM,EAAE,CAAC;IACxF,qFAAqF;IACrF,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC;CAChC;AAED,4EAA4E;AAC5E,wBAAgB,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAS5D;AA2BD;;;;GAIG;AACH,wBAAsB,mBAAmB,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,SAAS,EAAE,GAAG,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,OAAO,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC,CAOtI;AAED;;;;;;;;GAQG;AACH,wBAAsB,cAAc,CAAC,MAAM,EAAE,cAAc,EAAE,MAAM,EAAE,UAAU,EAAE,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,YAAY,CAAC,CAahI"}
1
+ {"version":3,"file":"run.d.ts","sourceRoot":"","sources":["../src/run.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAMH,OAAO,EAAoB,KAAK,UAAU,EAAE,KAAK,YAAY,EAAE,MAAM,kBAAkB,CAAC;AACxF,OAAO,EAAc,KAAK,cAAc,EAAE,KAAK,aAAa,EAAE,MAAM,2BAA2B,CAAC;AAChG,OAAO,EAA4B,KAAK,kBAAkB,EAAE,MAAM,cAAc,CAAC;AAEjF,6DAA6D;AAC7D,eAAO,MAAM,kBAAkB,QAAS,CAAC;AAEzC,4FAA4F;AAC5F,MAAM,MAAM,aAAa,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,cAAc,KAAK,OAAO,CAAC,aAAa,CAAC,CAAC;AAEjG,kCAAkC;AAClC,MAAM,WAAW,eAAgB,SAAQ,kBAAkB;IACzD,oFAAoF;IACpF,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACrC,2DAA2D;IAC3D,QAAQ,CAAC,IAAI,CAAC,EAAE,SAAS,MAAM,EAAE,GAAG,SAAS,CAAC;IAC9C,wDAAwD;IACxD,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACtC,iDAAiD;IACjD,QAAQ,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAC;IAC1C,4FAA4F;IAC5F,QAAQ,CAAC,MAAM,CAAC,EAAE,aAAa,GAAG,SAAS,CAAC;IAC5C;;;;OAIG;IACH,QAAQ,CAAC,UAAU,CAAC,EAAE,UAAU,GAAG,SAAS,CAAC;IAC7C,2FAA2F;IAC3F,QAAQ,CAAC,aAAa,CAAC,EAAE,CAAC,CAAC,OAAO,EAAE,YAAY,KAAK,IAAI,CAAC,GAAG,SAAS,CAAC;CACxE;AAqBD;;;;;;;;;;;;;GAaG;AACH,wBAAgB,eAAe,CAAC,QAAQ,EAAE,MAAM,GAAG,MAAM,EAAE,CAS1D;AA0BD,8BAA8B;AAC9B,MAAM,WAAW,YAAY;IAC3B,4CAA4C;IAC5C,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,6CAA6C;IAC7C,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,8CAA8C;IAC9C,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;CAC1B;AAED,kFAAkF;AAClF,MAAM,MAAM,UAAU,GAAG;IAAE,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAA;CAAE,GAAG;IAAE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAA;CAAE,GAAG;IAAE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAA;CAAE,CAAC;AAE1G,qFAAqF;AACrF,MAAM,WAAW,cAAc;IAC7B,oFAAoF;IACpF,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,oGAAoG;IACpG,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,gEAAgE;IAChE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,8FAA8F;IAC9F,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,SAAS,MAAM,EAAE,GAAG,MAAM,EAAE,CAAC;IACxF,qFAAqF;IACrF,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC;CAChC;AAED,4EAA4E;AAC5E,wBAAgB,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAS5D;AA2BD;;;;GAIG;AACH,wBAAsB,mBAAmB,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,SAAS,EAAE,GAAG,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,OAAO,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC,CAOtI;AAED;;;;;;;;;;;;;GAaG;AACH,wBAAsB,cAAc,CAAC,MAAM,EAAE,cAAc,EAAE,MAAM,EAAE,UAAU,EAAE,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,YAAY,CAAC,CAchI"}
package/dist/run.js CHANGED
@@ -5,15 +5,85 @@
5
5
  *
6
6
  * The browser starts through `runCommand` of `@procedures/shell`, not through a shell. A time limit
7
7
  * and the signal of the call stop its process tree.
8
+ *
9
+ * With the option `hostPolicy`, the run starts a policy proxy of `@procedures/http`. The browser
10
+ * sends each request through the proxy, and the proxy applies the host policy to each request.
11
+ * When the page is an `http:` or `https:` URL and the proxy gets no request, the run fails.
8
12
  */
9
13
  import { closeSync, copyFileSync, mkdirSync, mkdtempSync, openSync, readSync, rmSync, statSync, writeFileSync } from "node:fs";
10
14
  import { tmpdir } from "node:os";
11
15
  import { dirname, join, resolve } from "node:path";
12
16
  import { pathToFileURL } from "node:url";
17
+ import { startPolicyProxy } from "@procedures/http";
13
18
  import { runCommand } from "@procedures/shell/command";
14
19
  import { BROWSER_ENV, findBrowser } from "./browser.js";
15
20
  /** The default time limit of one render, in milliseconds. */
16
21
  export const DEFAULT_TIMEOUT_MS = 60_000;
22
+ /**
23
+ * The switches that the flags of the policy proxy set, and the switches that can undo these flags.
24
+ * With a host policy, the run refuses each of them in `args`.
25
+ */
26
+ const POLICY_SWITCHES = new Set([
27
+ "proxy-server",
28
+ "proxy-bypass-list",
29
+ "no-proxy-server",
30
+ "proxy-pac-url",
31
+ "proxy-auto-detect",
32
+ "use-system-proxy-resolver",
33
+ "winhttp-proxy-resolver",
34
+ "enable-quic",
35
+ "origin-to-force-quic-on",
36
+ "force-webrtc-ip-handling-policy",
37
+ "host-resolver-rules",
38
+ "host-rules",
39
+ ]);
40
+ /**
41
+ * This function gives the flags that send each request of the browser through the proxy at
42
+ * `proxyUrl`, and that close the paths around the proxy.
43
+ *
44
+ * - `--proxy-server` sends each `http:`, `https:`, `ws:` and `wss:` request to the proxy. The proxy
45
+ * resolves each name, not the browser.
46
+ * - `--proxy-bypass-list=<-loopback>` removes the default bypass of `localhost`, of the loopback
47
+ * addresses and of the link-local addresses. Thus these requests also go to the proxy.
48
+ * - `--disable-quic` stops the QUIC protocol, so no request goes over UDP around the proxy.
49
+ * - `--force-webrtc-ip-handling-policy=disable_non_proxied_udp` stops the UDP of WebRTC, and shows no
50
+ * local address.
51
+ * - `--host-resolver-rules` stops each DNS look-up of the browser. The only exception is the address
52
+ * of the proxy.
53
+ */
54
+ export function policyProxyArgs(proxyUrl) {
55
+ const { hostname } = new URL(proxyUrl);
56
+ return [
57
+ `--proxy-server=${proxyUrl}`,
58
+ "--proxy-bypass-list=<-loopback>",
59
+ "--disable-quic",
60
+ "--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
61
+ `--host-resolver-rules=MAP * ~NOTFOUND , EXCLUDE ${hostname}`,
62
+ ];
63
+ }
64
+ /**
65
+ * This function gives the name of the switch of a flag, or undefined for a value that is not a
66
+ * switch. The browser takes the prefixes `--`, `-` and, on Windows, `/`. Windows ignores the case of
67
+ * the name. Thus the function takes each prefix, and gives the name in lower case.
68
+ */
69
+ function switchName(arg) {
70
+ const match = /^(?:--|-|\/)([^=]+)/.exec(arg);
71
+ return match?.[1]?.toLowerCase();
72
+ }
73
+ /** This function throws when the options of the host policy do not agree with the other options. */
74
+ function checkPolicyOptions(options) {
75
+ if (options.hostPolicy === undefined) {
76
+ if (options.onHostRefusal !== undefined)
77
+ throw new Error("The option onHostRefusal needs the option hostPolicy");
78
+ return;
79
+ }
80
+ for (const arg of options.args ?? []) {
81
+ const name = switchName(arg);
82
+ if (name !== undefined && POLICY_SWITCHES.has(name)) {
83
+ throw new Error(`The option hostPolicy sets the proxy flags of the browser, so args cannot have the flag --${name}`);
84
+ }
85
+ }
86
+ }
17
87
  /** This function gives the first bytes of a file, at most `count` bytes. */
18
88
  export function readHead(path, count) {
19
89
  const fd = openSync(path, "r");
@@ -72,8 +142,14 @@ export async function withTemporaryFolder(prefix, folder, use) {
72
142
  * An existing output file changes only after the check of the new file. Thus a failed render keeps
73
143
  * an old output file. Without the option `output`, the file goes in a new temporary folder. This
74
144
  * folder stays after a success, and goes after a failure.
145
+ *
146
+ * With the option `hostPolicy`, the browser gets the flags of `policyProxyArgs` before the flags of
147
+ * `args`. The proxy stops at the end of the run, also after an error. When the page is an `http:`
148
+ * or `https:` URL and the proxy gets no request, the render fails. Then the requests of the page
149
+ * did not go through the policy.
75
150
  */
76
151
  export async function headlessRender(render, source, options) {
152
+ checkPolicyOptions(options);
77
153
  const browser = findBrowser(options);
78
154
  if (browser === undefined) {
79
155
  throw new Error(`No browser found. Install Microsoft Edge, Google Chrome or Chromium, or set ${BROWSER_ENV} to the browser program.`);
@@ -109,13 +185,22 @@ function renderOnce(render, source, options, browser, output) {
109
185
  mkdirSync(dirname(output), { recursive: true });
110
186
  const run = options.runner ?? runCommand;
111
187
  const timeout = options.timeout ?? DEFAULT_TIMEOUT_MS;
112
- const result = await run(browser, {
113
- args: render.argv(page, written, profile, options.args ?? []),
114
- timeout,
115
- signal: options.signal,
116
- maxOutputBytes: 64 * 1024,
117
- keep: "tail",
118
- });
188
+ const proxy = options.hostPolicy === undefined ? undefined : await startPolicyProxy({ policy: options.hostPolicy, onRefusal: options.onHostRefusal });
189
+ let result;
190
+ try {
191
+ // The flags of the proxy come before the extra flags. A "--" in the extra flags ends the switches of the browser
192
+ const extra = proxy === undefined ? (options.args ?? []) : [...policyProxyArgs(proxy.url), ...(options.args ?? [])];
193
+ result = await run(browser, {
194
+ args: render.argv(page, written, profile, extra),
195
+ timeout,
196
+ signal: options.signal,
197
+ maxOutputBytes: 64 * 1024,
198
+ keep: "tail",
199
+ });
200
+ }
201
+ finally {
202
+ await proxy?.close();
203
+ }
119
204
  if (result.error !== undefined)
120
205
  throw new Error(`The browser did not start: ${result.error}`);
121
206
  if (result.timedOut === true)
@@ -127,6 +212,11 @@ function renderOnce(render, source, options, browser, output) {
127
212
  const detail = lastLines(result.stderr || result.stdout);
128
213
  throw new Error(`The browser made no ${render.noun} (exit code ${result.exitCode})${detail ? `: ${detail}` : ""}`);
129
214
  }
215
+ // A Chrome or Edge policy (ProxySettings or ProxyMode) replaces the flag --proxy-server, and the browser gives no error.
216
+ // A web page sends at least one request, so a run with no request through the proxy is not a pass
217
+ if (proxy !== undefined && /^https?:/i.test(page) && proxy.requests === 0) {
218
+ throw new Error("The browser sent no request through the policy proxy, so the host policy did not apply. A Chrome or Edge policy (ProxySettings or ProxyMode) can replace the flag --proxy-server.");
219
+ }
130
220
  if (!render.isValid(written))
131
221
  throw new Error(`The file ${output} is not a ${render.noun}`);
132
222
  // The temporary folder goes after this function, so a copy is enough, also across two drives
package/dist/run.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"run.js","sourceRoot":"","sources":["../src/run.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,SAAS,EAAE,WAAW,EAAE,QAAQ,EAAE,QAAQ,EAAE,MAAM,EAAE,QAAQ,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AAC/H,OAAO,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AACjC,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACnD,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,UAAU,EAA2C,MAAM,2BAA2B,CAAC;AAChG,OAAO,EAAE,WAAW,EAAE,WAAW,EAA2B,MAAM,cAAc,CAAC;AAEjF,6DAA6D;AAC7D,MAAM,CAAC,MAAM,kBAAkB,GAAG,MAAM,CAAC;AA8CzC,4EAA4E;AAC5E,MAAM,UAAU,QAAQ,CAAC,IAAY,EAAE,KAAa;IAClD,MAAM,EAAE,GAAG,QAAQ,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;IAC/B,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QACjC,MAAM,IAAI,GAAG,QAAQ,CAAC,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;QAC7C,OAAO,IAAI,CAAC,QAAQ,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC;IAChC,CAAC;YAAS,CAAC;QACT,SAAS,CAAC,EAAE,CAAC,CAAC;IAChB,CAAC;AACH,CAAC;AAED,SAAS,QAAQ,CAAC,IAAY;IAC5B,IAAI,CAAC;QACH,OAAO,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC;IAC7B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,SAAS,CAAC;IACnB,CAAC;AACH,CAAC;AAED,SAAS,SAAS,CAAC,IAAY,EAAE,KAAK,GAAG,CAAC;IACxC,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC7D,CAAC;AAED;;;;GAIG;AACH,SAAS,YAAY,CAAC,IAAY;IAChC,IAAI,CAAC;QACH,MAAM,CAAC,IAAI,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC,EAAE,UAAU,EAAE,GAAG,EAAE,CAAC,CAAC;IACjF,CAAC;IAAC,MAAM,CAAC;QACP,yDAAyD;IAC3D,CAAC;AACH,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,mBAAmB,CAAI,MAAc,EAAE,MAA0B,EAAE,GAAiC;IACxH,MAAM,IAAI,GAAG,WAAW,CAAC,IAAI,CAAC,MAAM,IAAI,MAAM,EAAE,EAAE,MAAM,CAAC,CAAC,CAAC;IAC3D,IAAI,CAAC;QACH,OAAO,MAAM,GAAG,CAAC,IAAI,CAAC,CAAC;IACzB,CAAC;YAAS,CAAC;QACT,YAAY,CAAC,IAAI,CAAC,CAAC;IACrB,CAAC;AACH,CAAC;AAED;;;;;;;;GAQG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAAC,MAAsB,EAAE,MAAkB,EAAE,OAAwB;IACvG,MAAM,OAAO,GAAG,WAAW,CAAC,OAAO,CAAC,CAAC;IACrC,IAAI,OAAO,KAAK,SAAS,EAAE,CAAC;QAC1B,MAAM,IAAI,KAAK,CAAC,+EAA+E,WAAW,0BAA0B,CAAC,CAAC;IACxI,CAAC;IACD,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS;QAAE,OAAO,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC;IAC/G,MAAM,IAAI,GAAG,WAAW,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;IACxD,IAAI,CAAC;QACH,OAAO,MAAM,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;IACrF,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,YAAY,CAAC,IAAI,CAAC,CAAC;QACnB,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC;AAED,wHAAwH;AACxH,SAAS,UAAU,CAAC,MAAsB,EAAE,MAAkB,EAAE,OAAwB,EAAE,OAAe,EAAE,MAAc;IACvH,OAAO,mBAAmB,CAAC,MAAM,CAAC,MAAM,EAAE,SAAS,EAAE,KAAK,EAAE,IAAI,EAAE,EAAE;QAClE,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;QACtC,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;QACzC,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC;QACxC,IAAI,IAAY,CAAC;QACjB,IAAI,MAAM,IAAI,MAAM,EAAE,CAAC;YACrB,aAAa,CAAC,QAAQ,EAAE,MAAM,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;YAC7C,IAAI,GAAG,aAAa,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC;QACtC,CAAC;aAAM,IAAI,MAAM,IAAI,MAAM,EAAE,CAAC;YAC5B,IAAI,GAAG,aAAa,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC;QAClD,CAAC;aAAM,CAAC;YACN,IAAI,GAAG,MAAM,CAAC,GAAG,CAAC;QACpB,CAAC;QACD,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QAEhD,MAAM,GAAG,GAAG,OAAO,CAAC,MAAM,IAAI,UAAU,CAAC;QACzC,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,IAAI,kBAAkB,CAAC;QACtD,MAAM,MAAM,GAAG,MAAM,GAAG,CAAC,OAAO,EAAE;YAChC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,IAAI,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC,IAAI,IAAI,EAAE,CAAC;YAC7D,OAAO;YACP,MAAM,EAAE,OAAO,CAAC,MAAM;YACtB,cAAc,EAAE,EAAE,GAAG,IAAI;YACzB,IAAI,EAAE,MAAM;SACb,CAAC,CAAC;QACH,IAAI,MAAM,CAAC,KAAK,KAAK,SAAS;YAAE,MAAM,IAAI,KAAK,CAAC,8BAA8B,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC;QAC9F,IAAI,MAAM,CAAC,QAAQ,KAAK,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,iCAAiC,OAAO,KAAK,CAAC,CAAC;QAC7F,IAAI,MAAM,CAAC,OAAO,KAAK,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;QAEzE,MAAM,KAAK,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC;QAChC,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;YACvC,MAAM,MAAM,GAAG,SAAS,CAAC,MAAM,CAAC,MAAM,IAAI,MAAM,CAAC,MAAM,CAAC,CAAC;YACzD,MAAM,IAAI,KAAK,CAAC,uBAAuB,MAAM,CAAC,IAAI,eAAe,MAAM,CAAC,QAAQ,IAAI,MAAM,CAAC,CAAC,CAAC,KAAK,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QACrH,CAAC;QACD,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC;YAAE,MAAM,IAAI,KAAK,CAAC,YAAY,MAAM,aAAa,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;QAC5F,6FAA6F;QAC7F,YAAY,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC9B,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;IAC1C,CAAC,CAAC,CAAC;AACL,CAAC"}
1
+ {"version":3,"file":"run.js","sourceRoot":"","sources":["../src/run.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,SAAS,EAAE,WAAW,EAAE,QAAQ,EAAE,QAAQ,EAAE,MAAM,EAAE,QAAQ,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AAC/H,OAAO,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AACjC,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AACnD,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,gBAAgB,EAAsC,MAAM,kBAAkB,CAAC;AACxF,OAAO,EAAE,UAAU,EAA2C,MAAM,2BAA2B,CAAC;AAChG,OAAO,EAAE,WAAW,EAAE,WAAW,EAA2B,MAAM,cAAc,CAAC;AAEjF,6DAA6D;AAC7D,MAAM,CAAC,MAAM,kBAAkB,GAAG,MAAM,CAAC;AA2BzC;;;GAGG;AACH,MAAM,eAAe,GAAwB,IAAI,GAAG,CAAC;IACnD,cAAc;IACd,mBAAmB;IACnB,iBAAiB;IACjB,eAAe;IACf,mBAAmB;IACnB,2BAA2B;IAC3B,wBAAwB;IACxB,aAAa;IACb,yBAAyB;IACzB,iCAAiC;IACjC,qBAAqB;IACrB,YAAY;CACb,CAAC,CAAC;AAEH;;;;;;;;;;;;;GAaG;AACH,MAAM,UAAU,eAAe,CAAC,QAAgB;IAC9C,MAAM,EAAE,QAAQ,EAAE,GAAG,IAAI,GAAG,CAAC,QAAQ,CAAC,CAAC;IACvC,OAAO;QACL,kBAAkB,QAAQ,EAAE;QAC5B,iCAAiC;QACjC,gBAAgB;QAChB,2DAA2D;QAC3D,mDAAmD,QAAQ,EAAE;KAC9D,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,SAAS,UAAU,CAAC,GAAW;IAC7B,MAAM,KAAK,GAAG,qBAAqB,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAC9C,OAAO,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,CAAC;AACnC,CAAC;AAED,oGAAoG;AACpG,SAAS,kBAAkB,CAAC,OAAwB;IAClD,IAAI,OAAO,CAAC,UAAU,KAAK,SAAS,EAAE,CAAC;QACrC,IAAI,OAAO,CAAC,aAAa,KAAK,SAAS;YAAE,MAAM,IAAI,KAAK,CAAC,sDAAsD,CAAC,CAAC;QACjH,OAAO;IACT,CAAC;IACD,KAAK,MAAM,GAAG,IAAI,OAAO,CAAC,IAAI,IAAI,EAAE,EAAE,CAAC;QACrC,MAAM,IAAI,GAAG,UAAU,CAAC,GAAG,CAAC,CAAC;QAC7B,IAAI,IAAI,KAAK,SAAS,IAAI,eAAe,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;YACpD,MAAM,IAAI,KAAK,CAAC,6FAA6F,IAAI,EAAE,CAAC,CAAC;QACvH,CAAC;IACH,CAAC;AACH,CAAC;AA6BD,4EAA4E;AAC5E,MAAM,UAAU,QAAQ,CAAC,IAAY,EAAE,KAAa;IAClD,MAAM,EAAE,GAAG,QAAQ,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;IAC/B,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QACjC,MAAM,IAAI,GAAG,QAAQ,CAAC,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC;QAC7C,OAAO,IAAI,CAAC,QAAQ,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC;IAChC,CAAC;YAAS,CAAC;QACT,SAAS,CAAC,EAAE,CAAC,CAAC;IAChB,CAAC;AACH,CAAC;AAED,SAAS,QAAQ,CAAC,IAAY;IAC5B,IAAI,CAAC;QACH,OAAO,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC;IAC7B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,SAAS,CAAC;IACnB,CAAC;AACH,CAAC;AAED,SAAS,SAAS,CAAC,IAAY,EAAE,KAAK,GAAG,CAAC;IACxC,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC7D,CAAC;AAED;;;;GAIG;AACH,SAAS,YAAY,CAAC,IAAY;IAChC,IAAI,CAAC;QACH,MAAM,CAAC,IAAI,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC,EAAE,UAAU,EAAE,GAAG,EAAE,CAAC,CAAC;IACjF,CAAC;IAAC,MAAM,CAAC;QACP,yDAAyD;IAC3D,CAAC;AACH,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,mBAAmB,CAAI,MAAc,EAAE,MAA0B,EAAE,GAAiC;IACxH,MAAM,IAAI,GAAG,WAAW,CAAC,IAAI,CAAC,MAAM,IAAI,MAAM,EAAE,EAAE,MAAM,CAAC,CAAC,CAAC;IAC3D,IAAI,CAAC;QACH,OAAO,MAAM,GAAG,CAAC,IAAI,CAAC,CAAC;IACzB,CAAC;YAAS,CAAC;QACT,YAAY,CAAC,IAAI,CAAC,CAAC;IACrB,CAAC;AACH,CAAC;AAED;;;;;;;;;;;;;GAaG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAAC,MAAsB,EAAE,MAAkB,EAAE,OAAwB;IACvG,kBAAkB,CAAC,OAAO,CAAC,CAAC;IAC5B,MAAM,OAAO,GAAG,WAAW,CAAC,OAAO,CAAC,CAAC;IACrC,IAAI,OAAO,KAAK,SAAS,EAAE,CAAC;QAC1B,MAAM,IAAI,KAAK,CAAC,+EAA+E,WAAW,0BAA0B,CAAC,CAAC;IACxI,CAAC;IACD,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS;QAAE,OAAO,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC;IAC/G,MAAM,IAAI,GAAG,WAAW,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;IACxD,IAAI,CAAC;QACH,OAAO,MAAM,UAAU,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;IACrF,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,YAAY,CAAC,IAAI,CAAC,CAAC;QACnB,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC;AAED,wHAAwH;AACxH,SAAS,UAAU,CAAC,MAAsB,EAAE,MAAkB,EAAE,OAAwB,EAAE,OAAe,EAAE,MAAc;IACvH,OAAO,mBAAmB,CAAC,MAAM,CAAC,MAAM,EAAE,SAAS,EAAE,KAAK,EAAE,IAAI,EAAE,EAAE;QAClE,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;QACtC,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;QACzC,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC;QACxC,IAAI,IAAY,CAAC;QACjB,IAAI,MAAM,IAAI,MAAM,EAAE,CAAC;YACrB,aAAa,CAAC,QAAQ,EAAE,MAAM,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;YAC7C,IAAI,GAAG,aAAa,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC;QACtC,CAAC;aAAM,IAAI,MAAM,IAAI,MAAM,EAAE,CAAC;YAC5B,IAAI,GAAG,aAAa,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC;QAClD,CAAC;aAAM,CAAC;YACN,IAAI,GAAG,MAAM,CAAC,GAAG,CAAC;QACpB,CAAC;QACD,SAAS,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QAEhD,MAAM,GAAG,GAAG,OAAO,CAAC,MAAM,IAAI,UAAU,CAAC;QACzC,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,IAAI,kBAAkB,CAAC;QACtD,MAAM,KAAK,GAAG,OAAO,CAAC,UAAU,KAAK,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,MAAM,gBAAgB,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,UAAU,EAAE,SAAS,EAAE,OAAO,CAAC,aAAa,EAAE,CAAC,CAAC;QACtJ,IAAI,MAAqB,CAAC;QAC1B,IAAI,CAAC;YACH,iHAAiH;YACjH,MAAM,KAAK,GAAG,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,eAAe,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,GAAG,CAAC,OAAO,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC,CAAC;YACpH,MAAM,GAAG,MAAM,GAAG,CAAC,OAAO,EAAE;gBAC1B,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC,IAAI,EAAE,OAAO,EAAE,OAAO,EAAE,KAAK,CAAC;gBAChD,OAAO;gBACP,MAAM,EAAE,OAAO,CAAC,MAAM;gBACtB,cAAc,EAAE,EAAE,GAAG,IAAI;gBACzB,IAAI,EAAE,MAAM;aACb,CAAC,CAAC;QACL,CAAC;gBAAS,CAAC;YACT,MAAM,KAAK,EAAE,KAAK,EAAE,CAAC;QACvB,CAAC;QACD,IAAI,MAAM,CAAC,KAAK,KAAK,SAAS;YAAE,MAAM,IAAI,KAAK,CAAC,8BAA8B,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC;QAC9F,IAAI,MAAM,CAAC,QAAQ,KAAK,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,iCAAiC,OAAO,KAAK,CAAC,CAAC;QAC7F,IAAI,MAAM,CAAC,OAAO,KAAK,IAAI;YAAE,MAAM,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;QAEzE,MAAM,KAAK,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC;QAChC,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;YACvC,MAAM,MAAM,GAAG,SAAS,CAAC,MAAM,CAAC,MAAM,IAAI,MAAM,CAAC,MAAM,CAAC,CAAC;YACzD,MAAM,IAAI,KAAK,CAAC,uBAAuB,MAAM,CAAC,IAAI,eAAe,MAAM,CAAC,QAAQ,IAAI,MAAM,CAAC,CAAC,CAAC,KAAK,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QACrH,CAAC;QACD,yHAAyH;QACzH,kGAAkG;QAClG,IAAI,KAAK,KAAK,SAAS,IAAI,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,KAAK,CAAC,QAAQ,KAAK,CAAC,EAAE,CAAC;YAC1E,MAAM,IAAI,KAAK,CACb,mLAAmL,CACpL,CAAC;QACJ,CAAC;QACD,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC;YAAE,MAAM,IAAI,KAAK,CAAC,YAAY,MAAM,aAAa,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;QAC5F,6FAA6F;QAC7F,YAAY,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC9B,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;IAC1C,CAAC,CAAC,CAAC;AACL,CAAC"}
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "$schema": "https://json.schemastore.org/package",
3
3
  "name": "@procedures/browser",
4
- "version": "0.55.2",
4
+ "version": "0.56.0",
5
5
  "description": "Find a Chromium browser that is already installed (Edge, Chrome or Chromium), and render a page with it: a PDF file or a screenshot, with no browser download. Find and read the bookmarks files of the profiles of Chrome, Edge and Chromium.",
6
6
  "keywords": [
7
7
  "browser",
@@ -30,11 +30,13 @@
30
30
  "exports": {
31
31
  ".": {
32
32
  "types": "./dist/index.d.ts",
33
- "import": "./dist/index.js"
33
+ "import": "./dist/index.js",
34
+ "default": "./dist/index.js"
34
35
  },
35
36
  "./bookmarks": {
36
37
  "types": "./dist/bookmarks.d.ts",
37
- "import": "./dist/bookmarks.js"
38
+ "import": "./dist/bookmarks.js",
39
+ "default": "./dist/bookmarks.js"
38
40
  }
39
41
  },
40
42
  "files": [
@@ -55,7 +57,8 @@
55
57
  "provenance": true
56
58
  },
57
59
  "dependencies": {
58
- "@procedures/shell": "0.55.2"
60
+ "@procedures/http": "0.56.0",
61
+ "@procedures/shell": "0.56.0"
59
62
  },
60
63
  "devDependencies": {
61
64
  "typescript": "^7.0.2",
package/src/index.ts CHANGED
@@ -1,6 +1,6 @@
1
1
  // The package has no procedures. It gives functions only: the search, the run and the renders.
2
2
  export * from "./browser.js";
3
- export { DEFAULT_TIMEOUT_MS, type CommandRunner, type HeadlessOptions, type RenderResult } from "./run.js";
3
+ export { DEFAULT_TIMEOUT_MS, policyProxyArgs, type CommandRunner, type HeadlessOptions, type RenderResult } from "./run.js";
4
4
  export { isPdfFile, pdfArgs, renderPdf, type RenderPdfOptions, type RenderPdfResult } from "./pdf.js";
5
5
  export * from "./screenshot.js";
6
6
  // The bookmarks files of Chrome, Edge and Chromium: the parse (also on the browser entry ./bookmarks) and the search on the disk
package/src/run.ts CHANGED
@@ -5,12 +5,17 @@
5
5
  *
6
6
  * The browser starts through `runCommand` of `@procedures/shell`, not through a shell. A time limit
7
7
  * and the signal of the call stop its process tree.
8
+ *
9
+ * With the option `hostPolicy`, the run starts a policy proxy of `@procedures/http`. The browser
10
+ * sends each request through the proxy, and the proxy applies the host policy to each request.
11
+ * When the page is an `http:` or `https:` URL and the proxy gets no request, the run fails.
8
12
  */
9
13
 
10
14
  import { closeSync, copyFileSync, mkdirSync, mkdtempSync, openSync, readSync, rmSync, statSync, writeFileSync } from "node:fs";
11
15
  import { tmpdir } from "node:os";
12
16
  import { dirname, join, resolve } from "node:path";
13
17
  import { pathToFileURL } from "node:url";
18
+ import { startPolicyProxy, type HostPolicy, type ProxyRefusal } from "@procedures/http";
14
19
  import { runCommand, type CommandOptions, type CommandResult } from "@procedures/shell/command";
15
20
  import { BROWSER_ENV, findBrowser, type FindBrowserOptions } from "./browser.js";
16
21
 
@@ -32,6 +37,82 @@ export interface HeadlessOptions extends FindBrowserOptions {
32
37
  readonly signal?: AbortSignal | undefined;
33
38
  /** The function that runs the browser (default: `runCommand`). The tests give their own. */
34
39
  readonly runner?: CommandRunner | undefined;
40
+ /**
41
+ * The host policy of each request of the browser. With it, each request goes through a policy
42
+ * proxy, also each redirect and each resource of the page. The proxy gives the status 403 to
43
+ * each host that the policy refuses.
44
+ */
45
+ readonly hostPolicy?: HostPolicy | undefined;
46
+ /** This function gets each request that the host policy refused. It needs `hostPolicy`. */
47
+ readonly onHostRefusal?: ((refusal: ProxyRefusal) => void) | undefined;
48
+ }
49
+
50
+ /**
51
+ * The switches that the flags of the policy proxy set, and the switches that can undo these flags.
52
+ * With a host policy, the run refuses each of them in `args`.
53
+ */
54
+ const POLICY_SWITCHES: ReadonlySet<string> = new Set([
55
+ "proxy-server",
56
+ "proxy-bypass-list",
57
+ "no-proxy-server",
58
+ "proxy-pac-url",
59
+ "proxy-auto-detect",
60
+ "use-system-proxy-resolver",
61
+ "winhttp-proxy-resolver",
62
+ "enable-quic",
63
+ "origin-to-force-quic-on",
64
+ "force-webrtc-ip-handling-policy",
65
+ "host-resolver-rules",
66
+ "host-rules",
67
+ ]);
68
+
69
+ /**
70
+ * This function gives the flags that send each request of the browser through the proxy at
71
+ * `proxyUrl`, and that close the paths around the proxy.
72
+ *
73
+ * - `--proxy-server` sends each `http:`, `https:`, `ws:` and `wss:` request to the proxy. The proxy
74
+ * resolves each name, not the browser.
75
+ * - `--proxy-bypass-list=<-loopback>` removes the default bypass of `localhost`, of the loopback
76
+ * addresses and of the link-local addresses. Thus these requests also go to the proxy.
77
+ * - `--disable-quic` stops the QUIC protocol, so no request goes over UDP around the proxy.
78
+ * - `--force-webrtc-ip-handling-policy=disable_non_proxied_udp` stops the UDP of WebRTC, and shows no
79
+ * local address.
80
+ * - `--host-resolver-rules` stops each DNS look-up of the browser. The only exception is the address
81
+ * of the proxy.
82
+ */
83
+ export function policyProxyArgs(proxyUrl: string): string[] {
84
+ const { hostname } = new URL(proxyUrl);
85
+ return [
86
+ `--proxy-server=${proxyUrl}`,
87
+ "--proxy-bypass-list=<-loopback>",
88
+ "--disable-quic",
89
+ "--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
90
+ `--host-resolver-rules=MAP * ~NOTFOUND , EXCLUDE ${hostname}`,
91
+ ];
92
+ }
93
+
94
+ /**
95
+ * This function gives the name of the switch of a flag, or undefined for a value that is not a
96
+ * switch. The browser takes the prefixes `--`, `-` and, on Windows, `/`. Windows ignores the case of
97
+ * the name. Thus the function takes each prefix, and gives the name in lower case.
98
+ */
99
+ function switchName(arg: string): string | undefined {
100
+ const match = /^(?:--|-|\/)([^=]+)/.exec(arg);
101
+ return match?.[1]?.toLowerCase();
102
+ }
103
+
104
+ /** This function throws when the options of the host policy do not agree with the other options. */
105
+ function checkPolicyOptions(options: HeadlessOptions): void {
106
+ if (options.hostPolicy === undefined) {
107
+ if (options.onHostRefusal !== undefined) throw new Error("The option onHostRefusal needs the option hostPolicy");
108
+ return;
109
+ }
110
+ for (const arg of options.args ?? []) {
111
+ const name = switchName(arg);
112
+ if (name !== undefined && POLICY_SWITCHES.has(name)) {
113
+ throw new Error(`The option hostPolicy sets the proxy flags of the browser, so args cannot have the flag --${name}`);
114
+ }
115
+ }
35
116
  }
36
117
 
37
118
  /** The result of a render. */
@@ -120,8 +201,14 @@ export async function withTemporaryFolder<T>(prefix: string, folder: string | un
120
201
  * An existing output file changes only after the check of the new file. Thus a failed render keeps
121
202
  * an old output file. Without the option `output`, the file goes in a new temporary folder. This
122
203
  * folder stays after a success, and goes after a failure.
204
+ *
205
+ * With the option `hostPolicy`, the browser gets the flags of `policyProxyArgs` before the flags of
206
+ * `args`. The proxy stops at the end of the run, also after an error. When the page is an `http:`
207
+ * or `https:` URL and the proxy gets no request, the render fails. Then the requests of the page
208
+ * did not go through the policy.
123
209
  */
124
210
  export async function headlessRender(render: HeadlessRender, source: PageSource, options: HeadlessOptions): Promise<RenderResult> {
211
+ checkPolicyOptions(options);
125
212
  const browser = findBrowser(options);
126
213
  if (browser === undefined) {
127
214
  throw new Error(`No browser found. Install Microsoft Edge, Google Chrome or Chromium, or set ${BROWSER_ENV} to the browser program.`);
@@ -155,13 +242,21 @@ function renderOnce(render: HeadlessRender, source: PageSource, options: Headles
155
242
 
156
243
  const run = options.runner ?? runCommand;
157
244
  const timeout = options.timeout ?? DEFAULT_TIMEOUT_MS;
158
- const result = await run(browser, {
159
- args: render.argv(page, written, profile, options.args ?? []),
160
- timeout,
161
- signal: options.signal,
162
- maxOutputBytes: 64 * 1024,
163
- keep: "tail",
164
- });
245
+ const proxy = options.hostPolicy === undefined ? undefined : await startPolicyProxy({ policy: options.hostPolicy, onRefusal: options.onHostRefusal });
246
+ let result: CommandResult;
247
+ try {
248
+ // The flags of the proxy come before the extra flags. A "--" in the extra flags ends the switches of the browser
249
+ const extra = proxy === undefined ? (options.args ?? []) : [...policyProxyArgs(proxy.url), ...(options.args ?? [])];
250
+ result = await run(browser, {
251
+ args: render.argv(page, written, profile, extra),
252
+ timeout,
253
+ signal: options.signal,
254
+ maxOutputBytes: 64 * 1024,
255
+ keep: "tail",
256
+ });
257
+ } finally {
258
+ await proxy?.close();
259
+ }
165
260
  if (result.error !== undefined) throw new Error(`The browser did not start: ${result.error}`);
166
261
  if (result.timedOut === true) throw new Error(`The browser did not finish in ${timeout} ms`);
167
262
  if (result.aborted === true) throw new Error("The render was cancelled");
@@ -171,6 +266,13 @@ function renderOnce(render: HeadlessRender, source: PageSource, options: Headles
171
266
  const detail = lastLines(result.stderr || result.stdout);
172
267
  throw new Error(`The browser made no ${render.noun} (exit code ${result.exitCode})${detail ? `: ${detail}` : ""}`);
173
268
  }
269
+ // A Chrome or Edge policy (ProxySettings or ProxyMode) replaces the flag --proxy-server, and the browser gives no error.
270
+ // A web page sends at least one request, so a run with no request through the proxy is not a pass
271
+ if (proxy !== undefined && /^https?:/i.test(page) && proxy.requests === 0) {
272
+ throw new Error(
273
+ "The browser sent no request through the policy proxy, so the host policy did not apply. A Chrome or Edge policy (ProxySettings or ProxyMode) can replace the flag --proxy-server.",
274
+ );
275
+ }
174
276
  if (!render.isValid(written)) throw new Error(`The file ${output} is not a ${render.noun}`);
175
277
  // The temporary folder goes after this function, so a copy is enough, also across two drives
176
278
  copyFileSync(written, output);