bidi2pdf 0.1.20 → 0.1.22

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 523b63ba5c422c92dd950e0045261ebcd9b888d042d930fa312b87b4c94f3e31
4
- data.tar.gz: b6aeb5c8ba74244bd890434a0c46ed18c92afd827dfaa47a9520fca20039d676
3
+ metadata.gz: '008d5b0569ba4b7f8bbcd4092da72ce14e5d0e42589a97813f7c64c8bc6a0d9f'
4
+ data.tar.gz: 8266f91bfa7606d843bed392cee203060cc3b13d98435cc9a0c4a679b5041285
5
5
  SHA512:
6
- metadata.gz: 7dbae17f4831478c506f67a0b1e72ca4890b2716c183e5c25b40a68e9009c78f12d8414bae9f9fc4e5f6eab52e51d04c9c217ed0b48cd73f8d54af3d4feca9e6
7
- data.tar.gz: 000bc92f8ea29444b652f45e48ef9daca4f460e90d71cc731b84e825e32e20296b24df6161febc2efc214bd4fa74577079634a23260785235c4bf977ec49c53b
6
+ metadata.gz: 27ce2475fa8d30bc9c5447353767c6d2c06db59f57daa7fab7e7a337e26703cc1a0df04439d2b8b68455e5a872138775321a7f31988beb02dba47df0dd06d48d
7
+ data.tar.gz: acb216b0c656916ee8e8ea797b1fccd026c67fa6260ac5f230f1c44190ffb00455fd83efb5238826c97af0f0051cf68a7335d09e51e0aaa25b85c2efdc802290
data/CHANGELOG.md CHANGED
@@ -8,10 +8,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
8
8
 
9
9
  ## [Unreleased]
10
10
 
11
- [unreleased]: https://github.com/dieter-medium/bidi2pdf/compare/v0.1.20..HEAD
11
+ [unreleased]: https://github.com/dieter-medium/bidi2pdf/compare/v0.1.22..HEAD
12
12
 
13
13
  <!-- generated by git-cliff end -->
14
14
 
15
+ ## [0.1.22] - 2026-10-11
16
+
17
+ ### 🐛 Fixed
18
+ - Report a network-idle timeout as a timeout
19
+ - Let the network-idle wait time out cleanly
20
+
21
+ ### 🔄 Changed
22
+ - Merge pull request #164 from dieter-medium/fix/network-idle-timeout
23
+ - Merge pull request #163 from dieter-medium/feat/disable-script-execution
24
+
25
+ ### 🚀 Added
26
+ - Let a tab switch off script execution
27
+
28
+ ## [0.1.21] - 2026-10-09
29
+
30
+ ### 🐛 Fixed
31
+ - Define the word the hyphenation control prints
32
+ - Run image specs against the image from the PR
33
+ - Pin hyphenation data to a fixed language set
34
+ - Check hyphenation in the published image
35
+ - Move the hyphenation spec to its own PR
36
+ - Build chromedriver images one at a time
37
+ - Read watchdog settings as decimal numbers
38
+ - Close spares first and free their leases
39
+
40
+ ### 🔄 Changed
41
+ - Merge pull request #159 from dieter-medium/feat/hyphenation-spec
42
+ - Merge pull request #158 from dieter-medium/feat/hyphenation-data
43
+ - Merge pull request #157 from dieter-medium/fix/chromedriver-build-race
44
+ - Merge pull request #156 from dieter-medium/feat/chromedriver-watchdog
45
+ - Merge pull request #155 from dieter-medium/feat/shutdown-deadline
46
+
47
+ ### 🚀 Added
48
+ - Install hyphenation dictionaries in the images
49
+ - Restart a stuck chromedriver container
50
+ - Bound the warmer shutdown by a deadline
51
+
15
52
  ## [0.1.20] - 2026-09-30
16
53
 
17
54
  ### 🐛 Fixed
@@ -531,7 +568,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
531
568
 
532
569
  ### 🔄 Released
533
570
 
534
- - [unreleased](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.20..HEAD)
571
+ - [unreleased](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.22..HEAD)
572
+ - [0.1.22](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.21..v0.1.22)
573
+ - [0.1.21](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.20..v0.1.21)
535
574
  - [0.1.20](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.19..v0.1.20)
536
575
  - [0.1.19](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.18..v0.1.19)
537
576
  - [0.1.18](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.17..v0.1.18)
data/README.md CHANGED
@@ -347,6 +347,9 @@ tab.navigate_to "https://example.com/invoice/14432423"
347
347
  # or inline
348
348
  # tab.inject_style content: "body { background-color: red; }"
349
349
 
350
+ # Printing HTML you do not trust? Switch its scripts off before navigating:
351
+ # tab.disable_script_execution
352
+
350
353
  tab.wait_until_network_idle
351
354
  tab.print("my.pdf")
352
355
 
@@ -459,6 +462,67 @@ the box in either image, no extra install step needed.
459
462
  Chromium comes from Debian's packages at build time, so every build can carry a different
460
463
  Chromium - for a byte-exact pin use the digest (`docker buildx imagetools inspect <image:tag>`).
461
464
 
465
+ ### Hyphenation
466
+
467
+ The images (`chromedriver`, `bidi2pdf` and its slim variant) ship Chromium's hyphenation dictionaries
468
+ in `/usr/lib/chromium/hyphen-data/`, so CSS `hyphens: auto` breaks long words in the page's language
469
+ (`lang="de"` on the element or the document) - for exactly these languages, by the `lang` tag Blink
470
+ maps to each dictionary (case does not matter; Blink drops trailing subtags until one matches):
471
+
472
+ | `lang` | Dictionary |
473
+ |---|---|
474
+ | `de`, `de-DE`, `de-AT`, `de-CH` (and `de-DE-1901` - the region wins) | German, 1996 spelling (`de-1996`) |
475
+ | `de-1901` | German, traditional spelling |
476
+ | `de-CH-1901`, `de-LI-1901` | Swiss German, traditional spelling |
477
+ | `en`, `en-US` | US English |
478
+ | `en-GB`, `en-AU`, `en-IE`, ... (every other `en-*`) | British English |
479
+ | `fr`, `es`, `it`, `nl`, `pt` (with any region, e.g. `pt-BR`) | French, Spanish, Italian, Dutch, Portuguese |
480
+
481
+ The hyphenation spec prints one word in each of these, against a control without hyphenation.
482
+ Debian's Chromium has none, and the component it would download needs a profile that outlives the
483
+ session. `docker/install-hyphen-data.sh` fetches them at build time from Chromium's source, pinned by
484
+ the `HYPHEN_DATA_REF` build argument - deliberately independent of the Debian Chromium the image
485
+ installs, since the pattern data and its format change far more rarely; the hyphenation spec catches a
486
+ Chromium that stops reading them - with their licence notice (`LICENSE` next to them). Do not pass `--disable-component-update` in
487
+ `chrome_args`: Chromium then skips the dictionaries.
488
+
489
+ ### ChromeDriver health check and watchdog
490
+
491
+ The image has a Docker `HEALTHCHECK` (chromedriver's `/status`), so `docker ps` shows a
492
+ chromedriver that stopped answering as `unhealthy`. It does not see a Chrome that is wedged behind
493
+ a chromedriver that still answers - memory thrashing shows only to the watchdog. And Docker does
494
+ not restart a container for being unhealthy - only when its main process exits.
495
+
496
+ That is what the opt-in watchdog does. With `WATCHDOG_ENABLED=true` it checks every
497
+ `WATCHDOG_INTERVAL` seconds and, after `WATCHDOG_FAILURES` failed checks in a row, stops
498
+ chromedriver; the container exits non-zero and its restart policy starts a fresh one - any policy
499
+ but `no` (`--restart unless-stopped` or `on-failure`, `restart:` in Compose; Kamal sets
500
+ `unless-stopped` for accessories). Without one, a watchdog restart just stops the container. A
501
+ check fails when chromedriver does not answer `/status`, or when the
502
+ container's memory thrashes: Linux's pressure stall information (`memory.pressure`, "full") shows
503
+ every task stalled on memory for at least `WATCHDOG_MEMORY_PRESSURE` percent of the last 10 s - a
504
+ Chrome swapping at its memory limit, which answers no command but never crashes.
505
+
506
+ Renders still running on that Chrome fail when it restarts; a client with a sweeper retries them.
507
+
508
+ | Variable | Default | Description |
509
+ |----------------------------|----------------------------------|--------------------------------------------------------------|
510
+ | `WATCHDOG_ENABLED` | `false` | `true` starts the watchdog. |
511
+ | `WATCHDOG_INTERVAL` | `15` | Seconds between checks. |
512
+ | `WATCHDOG_FAILURES` | `3` | Failed checks in a row before the restart. |
513
+ | `WATCHDOG_STATUS_TIMEOUT` | `5` | Seconds `/status` may take. |
514
+ | `WATCHDOG_MEMORY_PRESSURE` | `50` | "full avg10" percent that fails a check; `0` turns it off. |
515
+ | `WATCHDOG_STOP_GRACE` | `10` | Seconds between stopping chromedriver (TERM) and killing it. |
516
+ | `WATCHDOG_PRESSURE_FILE` | `/sys/fs/cgroup/memory.pressure` | The PSI file; without one (cgroup v1, PSI off) only `/status` counts. |
517
+
518
+ All values are whole numbers (seconds, percent). A malformed value stops the watchdog from
519
+ starting - chromedriver keeps running unwatched, and the container log says why.
520
+
521
+ Picking `WATCHDOG_MEMORY_PRESSURE`: "full" counts only the time *every* task waited on memory, so it
522
+ stays near 0 while renders merely run; 50 % of the last 10 s is a container that is stuck, not busy.
523
+ Before lowering it, watch `/sys/fs/cgroup/memory.pressure` inside the container under your usual
524
+ render load and stay well above what you see there.
525
+
462
526
  ### Docker Compose
463
527
 
464
528
  ```bash
@@ -574,6 +638,7 @@ Bidi2pdf::SessionWarmer.shutdown
574
638
  | `registry_dir` | `Dir.tmpdir` | Where the session registry file lives - every process that should clean up after the others must share it. |
575
639
  | `sweeper` | `nil` | Remote only: settings for a [`ChromeSweeper`](#leaked-chrome-sessions-bidi2pdfchromesweeper) on the same chromedriver, e.g. `{ scope: :all, max_sessions: :auto, pids_limit: 1024, interval: 60 }`. The warmer's own sessions are never touched; `Bidi2pdf::SessionWarmer.sweep!` sweeps on demand. |
576
640
  | `retry_refused_sessions` | `true` | With a `sweeper`: when chromedriver refuses a new session, sweep under pressure and try once more. `false` leaves it to the caller. |
641
+ | `shutdown_timeout` | `10` | Seconds `shutdown` waits, all in all, for its background threads and for closing the warm sessions. `nil` waits for ever. |
577
642
 
578
643
  #### Leftover sessions on a shared chromedriver
579
644
 
@@ -595,6 +660,13 @@ With a `sweeper` configured, the warmer also sweeps in the background every `int
595
660
  and when chromedriver refuses a new session ("session not created" - typically a container out of
596
661
  room for another Chrome) it sweeps once and tries once more.
597
662
 
663
+ `shutdown` is bounded by `shutdown_timeout`. A Chrome that hangs or ran out of memory answers no
664
+ command, and closing each of its sessions would wait out every command timeout in turn - long
665
+ enough that a stopping server looks hung. At the deadline the warmer stops waiting, logs what is
666
+ still running and instruments `session_warmer.shutdown_timeout.bidi2pdf` (`timeout`, `pending`);
667
+ `shutdown` returns the same tally, e.g. `{ "spare" => 1 }`, and `{}` when everything finished.
668
+ Sessions it could not close stay in the registry, so a later sweep closes them.
669
+
598
670
  > **Security note:** an idle warm session is an open, unauthenticated automation endpoint
599
671
  > (chromedriver's port, Chrome's debugging port - loopback only for a local chromedriver) for as
600
672
  > long as it waits. `max_idle_age` bounds that window; keep it set unless the process runs somewhere
@@ -663,7 +735,7 @@ session is refused.
663
735
  | `pids_limit` | `nil` | The chromedriver container's pids limit (Docker counts threads). `:auto` without it means no limit. |
664
736
  | `pids_budget` | `0.8` | Share of `pids_limit` the Chrome sessions may use. |
665
737
  | `threads_per_session` | `110` | Threads one Chrome session uses. |
666
- | `interval` | `nil` | Seconds between background sweeps (`start`/`stop`). |
738
+ | `interval` | `nil` | Seconds between background sweeps (`start`/`stop`). `stop(timeout: 10)` waits that long for a sweep in progress, then stops waiting and returns `false` (`nil` waits for ever). |
667
739
  | `dry_run` | `false` | Report what would be closed, close nothing. |
668
740
  | `registry_dir` | `Dir.tmpdir`| The registry to read and update - same meaning as the warmer's setting. |
669
741
  | `own_sessions` | `-> { [] }` | A callable returning the caller's live session ids; they are never touched. |
data/docker/Dockerfile CHANGED
@@ -13,11 +13,23 @@ RUN apt-get update && apt-get upgrade -y &&\
13
13
  libatk-bridge2.0-0 \
14
14
  libgtk-3-0 \
15
15
  libdrm2 \
16
- curl \
16
+ curl ca-certificates \
17
17
  unzip \
18
18
  xvfb \
19
19
  && rm -rf /var/lib/apt/lists/*
20
20
 
21
+ # Chromium's hyphenation dictionaries, so that `hyphens: auto` works in the PDFs. Debian's chromium
22
+ # ships none and Chrome cannot download them here - see docker/install-hyphen-data.sh for the whole
23
+ # story. They go next to the chromium binary, where the component installer looks for a bundled copy.
24
+ # The patterns are data files and intentionally do not track the Debian Chromium installed above (the
25
+ # minikin .hyb format is unchanged since 2016, the files since 2023): update this ref when the pattern
26
+ # data itself changes. The hyphenation integration spec, run against the image's own Chromium, catches
27
+ # a Chromium that stops reading them.
28
+ ARG HYPHEN_DATA_REF=refs/tags/154.0.8037.92
29
+ COPY ./docker/install-hyphen-data.sh /usr/local/bin/install-hyphen-data.sh
30
+ RUN chmod +x /usr/local/bin/install-hyphen-data.sh && \
31
+ /usr/local/bin/install-hyphen-data.sh "${HYPHEN_DATA_REF}" /usr/lib/chromium/hyphen-data
32
+
21
33
  # Create a non-root user
22
34
  RUN groupadd -r appuser && useradd -r -g appuser -m -d /home/appuser appuser
23
35
 
@@ -18,7 +18,7 @@ RUN echo "deb http://deb.debian.org/debian bookworm contrib non-free" > /etc/apt
18
18
  libatk-bridge2.0-0 \
19
19
  libgtk-3-0 \
20
20
  libdrm2 \
21
- curl \
21
+ curl ca-certificates \
22
22
  unzip \
23
23
  xvfb \
24
24
  x11vnc \
@@ -34,11 +34,24 @@ RUN echo "deb http://deb.debian.org/debian bookworm contrib non-free" > /etc/apt
34
34
  && locale-gen en_US.UTF-8 \
35
35
  && rm -rf /var/lib/apt/lists/*
36
36
 
37
+ # Chromium's hyphenation dictionaries, so that `hyphens: auto` works in the PDFs. Debian's chromium
38
+ # ships none and Chrome cannot download them here - see docker/install-hyphen-data.sh for the whole
39
+ # story. They go next to the chromium binary, where the component installer looks for a bundled copy.
40
+ # The patterns are data files and intentionally do not track the Debian Chromium installed above (the
41
+ # minikin .hyb format is unchanged since 2016, the files since 2023): update this ref when the pattern
42
+ # data itself changes. The hyphenation integration spec, run against the image's own Chromium, catches
43
+ # a Chromium that stops reading them.
44
+ ARG HYPHEN_DATA_REF=refs/tags/154.0.8037.92
45
+ COPY ./docker/install-hyphen-data.sh /usr/local/bin/install-hyphen-data.sh
46
+ RUN chmod +x /usr/local/bin/install-hyphen-data.sh && \
47
+ /usr/local/bin/install-hyphen-data.sh "${HYPHEN_DATA_REF}" /usr/lib/chromium/hyphen-data
48
+
37
49
  # Create a non-root user
38
50
  RUN groupadd -r appuser && useradd -r -g appuser -G audio,video -m -d /home/appuser appuser
39
51
 
40
52
  COPY ./docker/entrypoint.sh /usr/local/bin/entrypoint.sh
41
- RUN chmod +x /usr/local/bin/entrypoint.sh
53
+ COPY ./docker/chromedriver-watchdog.sh /usr/local/bin/chromedriver-watchdog.sh
54
+ RUN chmod +x /usr/local/bin/entrypoint.sh /usr/local/bin/chromedriver-watchdog.sh
42
55
 
43
56
  # ARM compatibility workaround:
44
57
  # On ARM architectures (such as Apple Silicon), downloading chromedriver via automated scripts may fail or cause ELF binary errors,
@@ -69,4 +82,9 @@ EXPOSE ${CHROMEDRIVER_PORT}
69
82
  # VNC
70
83
  EXPOSE 5900
71
84
 
85
+ # Marks the container "unhealthy" in docker ps when chromedriver stops answering. Docker does not
86
+ # restart it for that - WATCHDOG_ENABLED=true does (chromedriver-watchdog.sh).
87
+ HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
88
+ CMD curl -fs -m 4 -o /dev/null "http://127.0.0.1:${CHROMEDRIVER_PORT}/status" || exit 1
89
+
72
90
  CMD ["/usr/local/bin/entrypoint.sh"]
@@ -50,13 +50,25 @@ RUN apt-get update && apt-get upgrade -y &&\
50
50
  libatk-bridge2.0-0 \
51
51
  libgtk-3-0 \
52
52
  libdrm2 \
53
- curl \
53
+ curl ca-certificates \
54
54
  unzip \
55
55
  xvfb \
56
56
  && rm -rf /var/lib/apt/lists/*
57
57
 
58
58
  COPY --from=builder /usr/local/bundle /usr/local/bundle
59
59
 
60
+ # Chromium's hyphenation dictionaries, so that `hyphens: auto` works in the PDFs. Debian's chromium
61
+ # ships none and Chrome cannot download them here - see docker/install-hyphen-data.sh for the whole
62
+ # story. They go next to the chromium binary, where the component installer looks for a bundled copy.
63
+ # The patterns are data files and intentionally do not track the Debian Chromium installed above (the
64
+ # minikin .hyb format is unchanged since 2016, the files since 2023): update this ref when the pattern
65
+ # data itself changes. The hyphenation integration spec, run against the image's own Chromium, catches
66
+ # a Chromium that stops reading them.
67
+ ARG HYPHEN_DATA_REF=refs/tags/154.0.8037.92
68
+ COPY ./docker/install-hyphen-data.sh /usr/local/bin/install-hyphen-data.sh
69
+ RUN chmod +x /usr/local/bin/install-hyphen-data.sh && \
70
+ /usr/local/bin/install-hyphen-data.sh "${HYPHEN_DATA_REF}" /usr/lib/chromium/hyphen-data
71
+
60
72
  # Create a non-root user
61
73
  RUN groupadd -r appuser && useradd -r -g appuser -m -d /home/appuser appuser
62
74
 
@@ -0,0 +1,115 @@
1
+ #!/bin/bash
2
+ # Restarts the chromedriver container when Chrome is in a state it does not recover from by itself:
3
+ # chromedriver no longer answers /status, or the container's memory is thrashing (PSI "full" - every
4
+ # task stalled on memory for that share of the last 10 s). It stops chromedriver, entrypoint.sh
5
+ # exits, and the container's restart policy (unless-stopped, also Kamal's for accessories) starts a
6
+ # fresh one. Docker itself never restarts a merely "unhealthy" container. Renders still running on
7
+ # that Chrome fail - the client's retry and sweeper handle that.
8
+ #
9
+ # Opt-in: entrypoint.sh starts it when WATCHDOG_ENABLED=true.
10
+ # Usage: chromedriver-watchdog.sh <chromedriver pid> [marker file]
11
+ # Before a restart it creates the marker file, so entrypoint.sh can exit non-zero and a restart
12
+ # policy of on-failure restarts the container too.
13
+ #
14
+ # WATCHDOG_INTERVAL seconds between checks (default 15)
15
+ # WATCHDOG_FAILURES failed checks in a row before restart (default 3)
16
+ # WATCHDOG_STATUS_TIMEOUT seconds /status may take (default 5)
17
+ # WATCHDOG_MEMORY_PRESSURE PSI full avg10 percent that fails a check, 0 = off (default 50)
18
+ # WATCHDOG_STOP_GRACE seconds between TERM and KILL (default 10)
19
+ # WATCHDOG_PRESSURE_FILE the cgroup's PSI file (default /sys/fs/cgroup/memory.pressure)
20
+ set -euo pipefail
21
+
22
+ pid="${1:-}"
23
+ marker="${2:-}"
24
+ port="${CHROMEDRIVER_PORT:-3000}"
25
+ interval="${WATCHDOG_INTERVAL:-15}"
26
+ failures="${WATCHDOG_FAILURES:-3}"
27
+ status_timeout="${WATCHDOG_STATUS_TIMEOUT:-5}"
28
+ pressure_limit="${WATCHDOG_MEMORY_PRESSURE:-50}"
29
+ stop_grace="${WATCHDOG_STOP_GRACE:-10}"
30
+ pressure_file="${WATCHDOG_PRESSURE_FILE:-/sys/fs/cgroup/memory.pressure}"
31
+ reason=""
32
+
33
+ log() {
34
+ printf 'chromedriver-watchdog: %s\n' "$*" >&2
35
+ }
36
+
37
+ # Prints the value as a plain decimal - "050" is 50, never bash's octal 40 - or fails closed on a
38
+ # malformed setting: the watchdog does not start (chromedriver keeps running).
39
+ decimal() {
40
+ local name="$1" value="$2" min="$3"
41
+ case "$value" in
42
+ '' | *[!0-9]*) ;;
43
+ *)
44
+ if ((10#$value >= min)); then
45
+ printf '%d\n' "$((10#$value))"
46
+ return
47
+ fi
48
+ ;;
49
+ esac
50
+ log "error: ${name} must be an integer >= ${min}, got '${value}' - not watching"
51
+ exit 1
52
+ }
53
+
54
+ pid="$(decimal "chromedriver pid" "$pid" 1)"
55
+ port="$(decimal CHROMEDRIVER_PORT "$port" 1)"
56
+ interval="$(decimal WATCHDOG_INTERVAL "$interval" 1)"
57
+ failures="$(decimal WATCHDOG_FAILURES "$failures" 1)"
58
+ status_timeout="$(decimal WATCHDOG_STATUS_TIMEOUT "$status_timeout" 1)"
59
+ pressure_limit="$(decimal WATCHDOG_MEMORY_PRESSURE "$pressure_limit" 0)"
60
+ stop_grace="$(decimal WATCHDOG_STOP_GRACE "$stop_grace" 1)"
61
+
62
+ # Whole percent of the last 10 s in which every task of this cgroup stalled on memory; 0 when the
63
+ # kernel offers no PSI file (cgroup v1, PSI disabled). Truncated, not rounded: 49.9 is 49, so a
64
+ # limit of 50 means at least 50.0.
65
+ memory_pressure() {
66
+ if [[ ! -r "$pressure_file" ]]; then
67
+ printf '0\n'
68
+ return
69
+ fi
70
+ awk '$1 == "full" { split($2, kv, "="); printf "%d\n", kv[2]; found = 1 } END { if (!found) print 0 }' "$pressure_file"
71
+ }
72
+
73
+ check() {
74
+ local pressure
75
+ if ! curl -fs -m "$status_timeout" -o /dev/null "http://127.0.0.1:${port}/status"; then
76
+ reason="chromedriver did not answer /status within ${status_timeout}s"
77
+ return 1
78
+ fi
79
+ if ((pressure_limit > 0)); then
80
+ pressure="$(memory_pressure)"
81
+ if ((pressure >= pressure_limit)); then
82
+ reason="memory pressure ${pressure}% (full avg10), limit ${pressure_limit}%"
83
+ return 1
84
+ fi
85
+ fi
86
+ }
87
+
88
+ restart() {
89
+ log "restarting: stopping chromedriver (pid ${pid}) so the container exits and its restart policy starts a fresh one"
90
+ if [[ -n "$marker" ]] && ! : >"$marker"; then
91
+ log "could not write ${marker} - the container may exit 0"
92
+ fi
93
+ kill -TERM "$pid" 2>/dev/null || true
94
+ for ((waited = 0; waited < stop_grace; waited++)); do
95
+ kill -0 "$pid" 2>/dev/null || exit 0
96
+ sleep 1
97
+ done
98
+ kill -KILL "$pid" 2>/dev/null || true
99
+ exit 0
100
+ }
101
+
102
+ log "watching chromedriver (pid ${pid}) every ${interval}s, restart after ${failures} failed checks"
103
+ failed=0
104
+ while kill -0 "$pid" 2>/dev/null; do
105
+ sleep "$interval"
106
+ if check; then
107
+ failed=0
108
+ continue
109
+ fi
110
+ failed=$((failed + 1))
111
+ log "check ${failed}/${failures} failed: ${reason}"
112
+ if ((failed >= failures)); then
113
+ restart
114
+ fi
115
+ done
data/docker/entrypoint.sh CHANGED
@@ -17,7 +17,7 @@ if [ "$ENABLE_XVFB" = "true" ]; then
17
17
 
18
18
  xauth generate :99 . trusted
19
19
 
20
- umask $old_umask
20
+ umask "$old_umask"
21
21
 
22
22
 
23
23
 
@@ -38,17 +38,45 @@ if [ "$ENABLE_VNC" = "true" ]; then
38
38
  old_umask=$(umask)
39
39
  umask 077
40
40
  mkdir -p /home/appuser/.vnc
41
- x11vnc -storepasswd $VNC_PASS /home/appuser/.vnc/passwd
42
- umask $old_umask
41
+ x11vnc -storepasswd "$VNC_PASS" /home/appuser/.vnc/passwd
42
+ umask "$old_umask"
43
43
  x11vnc -display WAIT:99 -xkb -noxrecord -noxfixes -noxdamage -forever -shared -noshm -usepw -rfbauth /home/appuser/.vnc/passwd &
44
44
  fi
45
45
 
46
46
  # DISPLAY=:99 /home/appuser/.webdrivers/chromedriver --port=33259 --whitelisted-ips="" --allowed-origins="*" --disable-dev-shm-usage --disable-gpu --verbose
47
- /home/appuser/.webdrivers/chromedriver --port=${CHROMEDRIVER_PORT} \
47
+ /home/appuser/.webdrivers/chromedriver --port="${CHROMEDRIVER_PORT}" \
48
48
  --allowed-ips="" \
49
49
  --allowed-origins="*" \
50
50
  --disable-dev-shm-usage \
51
51
  --disable-gpu \
52
- --user-data-dir=${USER_DATA_DIR} \
53
- --log-level=${CHROMEDRIVER_LOG_LEVEL:-INFO} \
54
- --readable-timestamp
52
+ --user-data-dir="${USER_DATA_DIR}" \
53
+ --log-level="${CHROMEDRIVER_LOG_LEVEL:-INFO}" \
54
+ --readable-timestamp &
55
+ chromedriver_pid=$!
56
+
57
+ # docker stop sends TERM to this script (PID 1): pass it on so chromedriver ends its sessions.
58
+ trap 'kill -TERM "$chromedriver_pid" 2>/dev/null' TERM INT
59
+
60
+ # Opt-in watchdog (see chromedriver-watchdog.sh): stops chromedriver when it no longer answers or
61
+ # memory thrashes, so this script exits and the restart policy starts a fresh container. It leaves
62
+ # a marker first, in a private temp dir, so the exit below is non-zero for any restart policy.
63
+ watchdog_marker=""
64
+ if [ "${WATCHDOG_ENABLED:-false}" = "true" ]; then
65
+ if watchdog_dir="$(mktemp -d)"; then
66
+ watchdog_marker="${watchdog_dir}/restarted"
67
+ fi
68
+ /usr/local/bin/chromedriver-watchdog.sh "$chromedriver_pid" "$watchdog_marker" &
69
+ fi
70
+
71
+ # A trapped signal interrupts wait before chromedriver has ended: wait again until it has.
72
+ status=0
73
+ wait "$chromedriver_pid" || status=$?
74
+ while kill -0 "$chromedriver_pid" 2>/dev/null; do
75
+ status=0
76
+ wait "$chromedriver_pid" || status=$?
77
+ done
78
+
79
+ if [ -n "$watchdog_marker" ] && [ -e "$watchdog_marker" ] && [ "$status" -eq 0 ]; then
80
+ status=1
81
+ fi
82
+ exit "$status"
@@ -0,0 +1,96 @@
1
+ #!/bin/bash
2
+ # Installs Chromium's hyphenation dictionaries so that `hyphens: auto` works in the PDFs.
3
+ #
4
+ # Why this exists: Chromium on Linux does not ship its hyphenation dictionaries in the binary. They
5
+ # come as a "Hyphenation" component which the component updater downloads at runtime into
6
+ # <user-data-dir>/hyphen-data/<version>/. In our containers that never sticks: chromedriver makes a
7
+ # fresh, throw-away user-data-dir for every session (a download would be gone with it), a sandboxed
8
+ # remote-chrome has no internet access to download from, and Debian's chromium package (chromium,
9
+ # chromium-common) ships no hyphen-data at all. So `hyphens: auto` silently renders as
10
+ # `hyphens: manual` - no automatic word breaks, however `lang` is set.
11
+ #
12
+ # How it works: before looking in the user-data-dir, Chromium's component installer looks for a
13
+ # "preinstalled" copy of every component next to its own binary (ComponentInstaller::StartRegistration ->
14
+ # FindPreinstallation, rooted at chrome::DIR_COMPONENTS = base::DIR_ASSETS = the directory of the chromium
15
+ # executable, /usr/lib/chromium on Debian). It expects <root>/hyphen-data/manifest.json with a valid
16
+ # "version" plus the hyph-<locale>.hyb files. That is exactly how Chrome for Testing bundles them
17
+ # (chrome/BUILD.gn -> third_party/hyphenation-patterns:bundle_hyphen_data), with the same bogus version
18
+ # "1.0.0.0" that is used below. The dictionaries themselves are the AOSP minikin .hyb files that live in
19
+ # chromium/src at third_party/hyphenation-patterns/hyb and are fetched from there, pinned to a Chromium tag
20
+ # (unchanged since 2023 - the de/en files of 151 and 154 are byte-identical). The pin intentionally does
21
+ # not follow the Debian Chromium the image installs: the patterns are data in a format unchanged since
22
+ # 2016, updated when the pattern data changes; the hyphenation integration spec, run against the
23
+ # image's own Chromium, catches a Chromium that stops reading them.
24
+ #
25
+ # The component is only registered by the full browser (chrome_browser_main.cc), never by the old
26
+ # headless shell - fine for Debian's chromium >= 132, where --headless is the new headless mode - and
27
+ # not when --disable-component-update is passed, which neither chromedriver nor bidi2pdf does. Never add it.
28
+ #
29
+ # Usage: install-hyphen-data.sh [chromium-ref] [target-dir]
30
+ set -euo pipefail
31
+
32
+ ref="${1:-${HYPHEN_DATA_REF:-refs/tags/154.0.8037.92}}"
33
+ target="${2:-/usr/lib/chromium/hyphen-data}"
34
+ source_dir="third_party/hyphenation-patterns"
35
+ archive_url="https://chromium.googlesource.com/chromium/src/+archive/${ref}/${source_dir}.tar.gz"
36
+
37
+ # The image's language contract: exactly these dictionaries are installed, and each must be in the
38
+ # download or the build fails - a renamed or dropped dictionary shows up at build time, not as a silent
39
+ # fallback to no hyphenation, and the image does not grow when Chromium adds languages. The archive
40
+ # carries ~50; add a language here (and to README "Hyphenation") to support it. Inside Blink "de" maps
41
+ # to de-1996, "en" tries en-gb, then en-us
42
+ # (third_party/blink/renderer/platform/text/hyphenation/hyphenation_minikin.cc, MapLocale).
43
+ dictionaries="hyph-de-1996.hyb hyph-de-1901.hyb hyph-de-ch-1901.hyb hyph-en-us.hyb hyph-en-gb.hyb hyph-fr.hyb hyph-es.hyb hyph-it.hyb hyph-nl.hyb hyph-pt.hyb"
44
+
45
+ workdir="$(mktemp -d)"
46
+ trap 'rm -rf "$workdir"' EXIT
47
+ mkdir -p "${workdir}/src/hyb"
48
+
49
+ # Gitiles exports a subtree as one tarball. If that is unreachable, fall back to the GitHub mirror of
50
+ # chromium/src (tags are mirrored), one file at a time, with the file list taken from the same BUILD.gn
51
+ # that Chrome for Testing bundles from.
52
+ fetch_from_gitiles() {
53
+ echo "Fetching Chromium hyphenation dictionaries from ${archive_url}"
54
+ curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/hyb.tar.gz" "${archive_url}" &&
55
+ tar -xzf "${workdir}/hyb.tar.gz" -C "${workdir}/src"
56
+ }
57
+
58
+ fetch_from_github() {
59
+ local raw="https://raw.githubusercontent.com/chromium/chromium/${ref#refs/tags/}/${source_dir}"
60
+ echo "Fetching Chromium hyphenation dictionaries from ${raw}"
61
+ curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/BUILD.gn" "${raw}/BUILD.gn"
62
+ curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/src/LICENSE" "${raw}/LICENSE"
63
+ grep -o '"hyb/hyph-[^"]*\.hyb"' "${workdir}/BUILD.gn" | tr -d '"' | sort -u | while read -r file; do
64
+ curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/src/${file}" "${raw}/${file}"
65
+ done
66
+ }
67
+
68
+ fetch_from_gitiles || { echo "gitiles download failed, trying the GitHub mirror" >&2; fetch_from_github; }
69
+
70
+ mkdir -p "${target}"
71
+ for file in ${dictionaries}; do
72
+ if [ ! -s "${workdir}/src/hyb/${file}" ]; then
73
+ echo "hyphenation dictionary ${file} missing from ${archive_url}" >&2
74
+ exit 1
75
+ fi
76
+ install -m 0644 "${workdir}/src/hyb/${file}" "${target}/${file}"
77
+ done
78
+ # The patterns are TeX hyphenation patterns under a mix of licenses; keep their notice with the data
79
+ # (Chromium's file covers every language, these included).
80
+ install -m 0644 "${workdir}/src/LICENSE" "${target}/LICENSE"
81
+
82
+ # Chromium's component installer only accepts the directory with a manifest carrying a valid version.
83
+ # 1.0.0.0 is what Chrome for Testing uses for its bundled copy: lower than anything the component
84
+ # updater would ever serve, and not all zeros (that means "no component").
85
+ cat > "${target}/manifest.json" <<'EOF'
86
+ {
87
+ "manifest_version": 2,
88
+ "name": "hyphens-data",
89
+ "version": "1.0.0.0"
90
+ }
91
+ EOF
92
+ chmod 0644 "${target}/manifest.json"
93
+ chmod 0755 "${target}"
94
+
95
+ dictionaries=("${target}"/*.hyb)
96
+ echo "Installed ${#dictionaries[@]} hyphenation dictionaries to ${target}"
@@ -334,15 +334,20 @@ module Bidi2pdf
334
334
  end
335
335
  end
336
336
 
337
- # Waits until the network is idle in the browser tab.
337
+ # Waits until the network is idle in the browser tab. A timeout does not raise: the
338
+ # network_idle.bidi2pdf event then carries +timed_out: true+, so subscribers never report a busy
339
+ # page as idle.
338
340
  #
339
341
  # @param [Integer] timeout The timeout duration in seconds. Defaults to 10.
340
342
  # @param [Float] poll_interval The polling interval in seconds. Defaults to 0.1.
343
+ # @return [Symbol] :idle or :timeout
341
344
  def wait_until_network_idle(timeout: 10, poll_interval: 0.1)
342
345
  Bidi2pdf.notification_service.instrument("network_idle.bidi2pdf") do |instrumentation_payload|
343
- network_events.wait_until_network_idle(timeout: timeout, poll_interval: poll_interval)
346
+ status = network_events.wait_until_network_idle(timeout: timeout, poll_interval: poll_interval)
344
347
 
345
348
  instrumentation_payload[:requests] = network_events.all_events.dup
349
+ instrumentation_payload[:timed_out] = status == :timeout
350
+ status
346
351
  end
347
352
  end
348
353
 
@@ -496,6 +501,18 @@ module Bidi2pdf
496
501
  client.send_cmd_and_wait(cmd) { |response| response }
497
502
  end
498
503
 
504
+ # Switches JavaScript off (or on again) for this tab, through Chrome's own
505
+ # Emulation.setScriptExecutionDisabled (CDP, via goog:cdp.sendCommand). Call it before
506
+ # navigating: it holds for every later navigation in the tab, so a page printed from untrusted
507
+ # HTML runs none of its scripts.
508
+ #
509
+ # @param [Boolean] disabled Defaults to true.
510
+ def disable_script_execution(disabled: true)
511
+ cmd = Commands::CdpSendCommand.new(method: "Emulation.setScriptExecutionDisabled", params: { value: disabled },
512
+ session: cdp_session)
513
+ client.send_cmd_and_wait(cmd) { |response| response }
514
+ end
515
+
499
516
  private
500
517
 
501
518
  def build_command_and_extractor(print_options)
@@ -95,24 +95,33 @@ module Bidi2pdf
95
95
  end
96
96
  end
97
97
 
98
+ # Waits until no request is in flight, or until +timeout+ seconds passed.
99
+ #
100
+ # @return [Symbol] :idle when every request completed, :timeout when some were still open -
101
+ # it never raises for a timeout, so a caller decides whether a busy page is good enough.
98
102
  def wait_until_network_idle(timeout: 10, poll_interval: 0.01)
99
103
  start_time = Time.now
100
104
 
101
105
  loop do
102
106
  unless events.values.any?(&:in_progress?)
103
107
  Bidi2pdf.logger.debug "✅ All network events completed."
104
- break
108
+ return :idle
105
109
  end
106
110
 
107
111
  if Time.now - start_time > timeout
108
- Bidi2pdf.logger.warn "⏰ Timeout while waiting for network events to complete. Still in progress: #{in_progress.map(&:id)}"
109
- # rubocop:enable Layout/LineLength
110
- break
112
+ log_idle_timeout
113
+ return :timeout
111
114
  end
112
115
 
113
116
  sleep(poll_interval)
114
117
  end
115
118
  end
119
+
120
+ # Names the requests still open when the wait gives up - it carries on, it does not raise.
121
+ def log_idle_timeout
122
+ in_progress = events.values.select(&:in_progress?)
123
+ Bidi2pdf.logger.warn "⏰ Timeout while waiting for network events to complete. Still in progress: #{in_progress.map(&:id)}"
124
+ end
116
125
  end
117
126
  end
118
127
  end
@@ -9,6 +9,7 @@ module Bidi2pdf
9
9
  DEFAULT_PIDS_BUDGET = 0.8
10
10
  DEFAULT_THREADS_PER_SESSION = 110
11
11
  DEFAULT_LEASE_TTL = SessionRegistry::DEFAULT_LEASE_TTL
12
+ DEFAULT_STOP_TIMEOUT = 10
12
13
 
13
14
  # What a ChromeSweeper closes and when - validated on construction; an unknown setting raises
14
15
  # ArgumentError.
@@ -167,21 +167,32 @@ module Bidi2pdf
167
167
  end
168
168
  end
169
169
 
170
- # Starts the periodic sweep thread (needs an +interval+). Idempotent.
170
+ # Starts the periodic sweep thread (needs an +interval+). Idempotent. Each thread has its own
171
+ # wakeup queue, so one that #stop stopped waiting for can never take a later thread's :stop.
172
+ # After such a #stop, a new thread can sweep while the abandoned sweep is still running: the
173
+ # lock file lets only one of them sweep at a time, the other's sweep is skipped.
171
174
  def start
172
175
  raise Bidi2pdf::InvalidConfigError, "chrome_sweeper: start needs an interval" unless interval
176
+ return self if @thread
173
177
 
174
- @thread ||= Thread.new { sweep!(reason: :periodic) until @wakeup.pop(timeout: interval) == :stop }
178
+ wakeup = @wakeup = Thread::Queue.new
179
+ @thread = Thread.new { sweep!(reason: :periodic) until wakeup.pop(timeout: interval) == :stop }
175
180
  self
176
181
  end
177
182
 
178
- # Stops the periodic sweep thread and waits for a sweep in progress to finish.
179
- def stop
180
- return unless @thread
183
+ # Stops the periodic sweep thread and waits up to +timeout+ seconds (nil: for ever) for a sweep
184
+ # in progress to finish - one inspecting sessions of a Chrome that no longer answers can take
185
+ # far longer. A sweep still running then is left to finish on its own; the thread ends after it.
186
+ #
187
+ # @return [Boolean] true when the thread has ended.
188
+ def stop(timeout: DEFAULT_STOP_TIMEOUT)
189
+ return true unless @thread
181
190
 
182
191
  @wakeup << :stop
183
- @thread.join
192
+ stopped = !@thread.join(timeout).nil?
193
+ Bidi2pdf.logger.warn "chrome_sweeper: a sweep of #{@session_url} still running after #{timeout}s - not waiting for it" unless stopped
184
194
  @thread = nil
195
+ stopped
185
196
  end
186
197
 
187
198
  private
@@ -53,8 +53,9 @@ module Bidi2pdf
53
53
  write(event: "page_loaded")
54
54
  end
55
55
 
56
- def on_network_idle(_event)
57
- write(event: "network_idle")
56
+ # A wait that timed out is its own event: a consumer reading "network_idle" can trust it.
57
+ def on_network_idle(event)
58
+ write(event: event.payload[:timed_out] ? "network_idle_timeout" : "network_idle")
58
59
  end
59
60
 
60
61
  def on_console(event)
@@ -61,7 +61,7 @@ module Bidi2pdf
61
61
  def network_idle(event)
62
62
  return unless logger.info?
63
63
 
64
- requests = event.payload[:requests]
64
+ requests = event.payload[:requests] || []
65
65
  transfered = requests.map { |request| request.bytes_received || 0 }.sum
66
66
  status_counts = requests
67
67
  .group_by { |evt| evt.http_status_code || 0 }
@@ -70,8 +70,12 @@ module Bidi2pdf
70
70
  .map { |code, count| "#{code}: #{count}" }
71
71
  .join(", ")
72
72
 
73
- logger.info "Network was idle after #{event.duration.round(1)}ms, #{requests.size} requests, " \
74
- "transferred #{transfered} bytes (status codes: #{status_counts})"
73
+ summary = "#{requests.size} requests, transferred #{transfered} bytes (status codes: #{status_counts})"
74
+ if event.payload[:timed_out]
75
+ logger.warn "Network was still busy after #{event.duration.round(1)}ms (timed out), #{summary}"
76
+ else
77
+ logger.info "Network was idle after #{event.duration.round(1)}ms, #{summary}"
78
+ end
75
79
  end
76
80
 
77
81
  # rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
@@ -196,7 +196,8 @@ module Bidi2pdf
196
196
  "properties" => {
197
197
  "schema_version" => { "const" => 1 },
198
198
  "t_ms" => { "type" => "integer", "description" => "milliseconds since the render started" },
199
- "event" => { "type" => "string", "enum" => %w[navigate page_loaded network_idle console print screenshot result] },
199
+ "event" => { "type" => "string", "enum" => %w[navigate page_loaded network_idle network_idle_timeout console print screenshot result],
200
+ "description" => "network_idle_timeout: the wait for an idle network gave up with requests still open" },
200
201
  "url" => { "type" => "string", "description" => "present on a navigate event" },
201
202
  "level" => { "type" => "string", "description" => "present on a console event" },
202
203
  "text" => { "type" => "string", "description" => "present on a console event" },
@@ -72,7 +72,15 @@ module Bidi2pdf
72
72
  # pressure and try once more. Default true; false leaves a refused session to the caller.
73
73
  attr_accessor :retry_refused_sessions
74
74
 
75
+ # @return [Numeric, nil] Seconds #shutdown waits, all in all, for its background threads and
76
+ # for closing the warm spares. A Chrome that hangs or ran out of memory answers no command,
77
+ # and every close would wait out its own timeout - an exiting process must not hang on that.
78
+ # What is still running then is left behind (a warning, +session_warmer.shutdown_timeout+);
79
+ # its sessions stay in the registry and a later sweep closes them. +nil+ waits for ever.
80
+ attr_accessor :shutdown_timeout
81
+
75
82
  DEFAULT_MAX_IDLE_AGE = 300
83
+ DEFAULT_SHUTDOWN_TIMEOUT = 10
76
84
 
77
85
  def initialize
78
86
  @size = 1
@@ -85,6 +93,7 @@ module Bidi2pdf
85
93
  @registry_dir = nil
86
94
  @sweeper = nil
87
95
  @retry_refused_sessions = true
96
+ @shutdown_timeout = DEFAULT_SHUTDOWN_TIMEOUT
88
97
  end
89
98
 
90
99
  # The orphan age in seconds, or nil when the sweep is off (also when +:auto+ has no
@@ -93,12 +102,20 @@ module Bidi2pdf
93
102
  orphan_age == :auto ? max_idle_age && (max_idle_age * 2) : orphan_age
94
103
  end
95
104
 
105
+ # nil (no limit) or a positive number of seconds - for shutdown_timeout and #shutdown's argument.
106
+ def self.validate_timeout!(name, value)
107
+ return if value.nil? || (value.is_a?(Numeric) && value.positive?)
108
+
109
+ raise ArgumentError, "#{name} must be nil or a positive number of seconds, got #{value.inspect}"
110
+ end
111
+
96
112
  # @raise [ArgumentError] if a setting can't be honored - checked once, at construction.
97
113
  def validate!
98
114
  validate_size!
99
115
  validate_max_idle_age!
100
116
  validate_orphan_age!
101
117
  validate_sweeper!
118
+ validate_shutdown_timeout!
102
119
  end
103
120
 
104
121
  private
@@ -121,6 +138,10 @@ module Bidi2pdf
121
138
  raise ArgumentError, "orphan_age must be :auto, nil or a positive number of seconds, got #{orphan_age.inspect}"
122
139
  end
123
140
 
141
+ def validate_shutdown_timeout!
142
+ Configuration.validate_timeout!("shutdown_timeout", shutdown_timeout)
143
+ end
144
+
124
145
  def validate_sweeper!
125
146
  return if sweeper.nil? || sweeper.is_a?(Hash)
126
147
 
@@ -273,20 +294,78 @@ module Bidi2pdf
273
294
  # finish (each of those, seeing @shutdown, retires its own result instead of stashing it - see
274
295
  # #stash_or_retire). A subsequent #with_tab still works - it just falls back to a synchronous
275
296
  # slot, since checkout never depends on a warm one being there.
276
- def shutdown
277
- spares, threads = @mutex.synchronize do
297
+ #
298
+ # All of that within +timeout+ seconds together (Configuration#shutdown_timeout; nil waits for
299
+ # ever). The spares start closing first, side by side, so a hung sweep cannot use up their share
300
+ # of the deadline. Whatever is still running at the deadline is left to finish on its own, or to
301
+ # die with the process; a spare still closing loses its lease (it keeps its registry entry), so
302
+ # a sweeper can take its session even while this process lives on.
303
+ #
304
+ # @return [Hash{String => Integer}] what was still running at the deadline ("spare" => 1, ...) -
305
+ # empty when everything finished in time.
306
+ # @raise [ArgumentError] for a +timeout+ that is neither nil nor a positive number.
307
+ def shutdown(timeout: @config.shutdown_timeout)
308
+ Configuration.validate_timeout!("timeout", timeout)
309
+ spares, threads = stop_accepting
310
+ deadline = timeout && (now + timeout)
311
+ closers = retire_all(spares)
312
+ pending = (stop_background(threads, deadline) + abandoned_spares(closers, deadline)).tally
313
+ warn_abandoned(pending, timeout) unless pending.empty?
314
+ pending
315
+ end
316
+
317
+ private
318
+
319
+ def stop_accepting
320
+ @mutex.synchronize do
278
321
  @shutdown = true
279
322
  [@available.dup.tap { @available.clear }, @replenish_threads.dup.tap { @replenish_threads.clear }]
280
323
  end
324
+ end
281
325
 
326
+ # The labels of whatever is still running at the deadline: reaper, sweeper, replenishments.
327
+ def stop_background(threads, deadline)
282
328
  @reaper_wakeup << :stop
283
- @reaper&.join
284
- @sweeper&.stop
285
- threads.each(&:join)
286
- spares.each { |slot| retire(slot) }
329
+ pending = unfinished([@reaper].compact, deadline, "reaper")
330
+ pending << "sweeper" unless @sweeper.nil? || @sweeper.stop(timeout: remaining(deadline))
331
+ pending + unfinished(threads, deadline, "replenishment")
287
332
  end
288
333
 
289
- private
334
+ def remaining(deadline)
335
+ deadline && [deadline - now, 0].max
336
+ end
337
+
338
+ # Thread#join returns nil when the time ran out first.
339
+ def unfinished(threads, deadline, label)
340
+ threads.reject { |thread| thread.join(remaining(deadline)) }.map { label }
341
+ end
342
+
343
+ # [slot, closing thread] pairs.
344
+ def retire_all(slots)
345
+ slots.map do |slot|
346
+ closer = Thread.new do
347
+ retire(slot)
348
+ rescue StandardError => e
349
+ Bidi2pdf.logger.warn "session_warmer: retiring a spare failed: #{e.message}"
350
+ end
351
+ [slot, closer]
352
+ end
353
+ end
354
+
355
+ # Spares still closing at the deadline, with their lease released: the heartbeat would otherwise
356
+ # keep renewing it, and no sweeper takes a leased session.
357
+ def abandoned_spares(closers, deadline)
358
+ closers.reject { |_, closer| closer.join(remaining(deadline)) }.map do |slot, _|
359
+ @registry&.release(session_id_of(slot))
360
+ "spare"
361
+ end
362
+ end
363
+
364
+ # Only called with something pending, which needs a deadline - so +timeout+ is a number here.
365
+ def warn_abandoned(pending, timeout)
366
+ Bidi2pdf.logger.warn "session_warmer: shutdown gave up after #{timeout}s, still running: #{pending.map { |what, count| "#{count} #{what}" }.join(", ")}"
367
+ Bidi2pdf.notification_service.instrument("session_warmer.shutdown_timeout.bidi2pdf", { timeout: timeout, pending: pending })
368
+ end
290
369
 
291
370
  # Fail-fast on purpose (a Chrome that can't start at boot should be loud), but not leaky: if slot
292
371
  # N fails, no instance is returned to own slots 1..N-1, so they are retired here first.
@@ -10,6 +10,14 @@ module Bidi2pdf
10
10
  # then sits silently in the pull for many minutes.
11
11
  DEFAULT_IMAGE = "dieters877565/chromedriver:latest"
12
12
 
13
+ # BIDI2PDF_BUILD_CHROMEDRIVER_IMAGE=true: build the image from the checkout (build_dir,
14
+ # docker_file) and run that, so a change to the image is tested before it is published -
15
+ # bidi2pdf's own CI sets it. Unset, the published image is pulled. Testcontainers' #start
16
+ # always pulls, so a locally built tag cannot go through it; #start_local_image builds instead.
17
+ def self.build_locally?(env = ENV)
18
+ env["BIDI2PDF_BUILD_CHROMEDRIVER_IMAGE"] == "true"
19
+ end
20
+
13
21
  attr_reader :docker_file, :build_dir
14
22
 
15
23
  def initialize(image = DEFAULT_IMAGE, **options)
@@ -132,7 +132,7 @@ def start_chromedriver_container(build_dir:, mounts:, shared_network:, chromedri
132
132
 
133
133
  container.with_filesystem_binds(mounts) if mounts&.any?
134
134
 
135
- container.start
135
+ ChromedriverTestcontainer.build_locally? ? container.start_local_image : container.start
136
136
 
137
137
  container
138
138
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Bidi2pdf
4
- VERSION = "0.1.20"
4
+ VERSION = "0.1.22"
5
5
  end
@@ -169,10 +169,13 @@ module Bidi2pdf
169
169
  # @return [Object] The result from the style creation promise.
170
170
  def inject_style: (?url: untyped?, ?content: untyped?, ?id: untyped?) -> untyped
171
171
 
172
- # Waits until the network is idle in the browser tab.
172
+ # Waits until the network is idle in the browser tab. A timeout does not raise: the
173
+ # network_idle.bidi2pdf event then carries +timed_out: true+, so subscribers never report a busy
174
+ # page as idle.
173
175
  #
174
176
  # @param [Integer] timeout The timeout duration in seconds. Defaults to 10.
175
177
  # @param [Float] poll_interval The polling interval in seconds. Defaults to 0.1.
178
+ # @return [Symbol] :idle or :timeout
176
179
  def wait_until_network_idle: (?timeout: ::Integer, ?poll_interval: ::Float) -> untyped
177
180
 
178
181
  # Waits until the page is fully loaded in the browser tab.
@@ -227,6 +230,14 @@ module Bidi2pdf
227
230
  # @param [Float, nil] device_pixel_ratio Defaults to nil (browser default).
228
231
  def set_viewport: (width: untyped, height: untyped, ?device_pixel_ratio: untyped?) -> untyped
229
232
 
233
+ # Switches JavaScript off (or on again) for this tab, through Chrome's own
234
+ # Emulation.setScriptExecutionDisabled (CDP, via goog:cdp.sendCommand). Call it before
235
+ # navigating: it holds for every later navigation in the tab, so a page printed from untrusted
236
+ # HTML runs none of its scripts.
237
+ #
238
+ # @param [Boolean] disabled Defaults to true.
239
+ def disable_script_execution: (?disabled: bool) -> untyped
240
+
230
241
  private
231
242
 
232
243
  def build_command_and_extractor: (untyped print_options) -> ::Array[untyped]
@@ -24,7 +24,14 @@ module Bidi2pdf
24
24
 
25
25
  def log_network_traffic: (?format: ::Symbol) -> untyped
26
26
 
27
- def wait_until_network_idle: (?timeout: ::Integer, ?poll_interval: ::Float) -> untyped
27
+ # Waits until no request is in flight, or until +timeout+ seconds passed.
28
+ #
29
+ # @return [Symbol] :idle when every request completed, :timeout when some were still open -
30
+ # it never raises for a timeout, so a caller decides whether a busy page is good enough.
31
+ def wait_until_network_idle: (?timeout: ::Integer, ?poll_interval: ::Float) -> (:idle | :timeout | untyped)
32
+
33
+ # Names the requests still open when the wait gives up - it carries on, it does not raise.
34
+ def log_idle_timeout: () -> untyped
28
35
  end
29
36
  end
30
37
  end
@@ -14,6 +14,8 @@ module Bidi2pdf
14
14
 
15
15
  DEFAULT_LEASE_TTL: untyped
16
16
 
17
+ DEFAULT_STOP_TIMEOUT: 10
18
+
17
19
  # What a ChromeSweeper closes and when - validated on construction; an unknown setting raises
18
20
  # ArgumentError.
19
21
  #
@@ -133,11 +133,18 @@ module Bidi2pdf
133
133
  # @return [Result]
134
134
  def sweep!: (?reason: ::Symbol, ?check_interval: untyped?, ?pressure: bool) -> untyped
135
135
 
136
- # Starts the periodic sweep thread (needs an +interval+). Idempotent.
136
+ # Starts the periodic sweep thread (needs an +interval+). Idempotent. Each thread has its own
137
+ # wakeup queue, so one that #stop stopped waiting for can never take a later thread's :stop.
138
+ # After such a #stop, a new thread can sweep while the abandoned sweep is still running: the
139
+ # lock file lets only one of them sweep at a time, the other's sweep is skipped.
137
140
  def start: () -> self
138
141
 
139
- # Stops the periodic sweep thread and waits for a sweep in progress to finish.
140
- def stop: () -> (nil | untyped)
142
+ # Stops the periodic sweep thread and waits up to +timeout+ seconds (nil: for ever) for a sweep
143
+ # in progress to finish - one inspecting sessions of a Chrome that no longer answers can take
144
+ # far longer. A sweep still running then is left to finish on its own; the thread ends after it.
145
+ #
146
+ # @return [Boolean] true when the thread has ended.
147
+ def stop: (?timeout: untyped) -> (true | untyped)
141
148
 
142
149
  private
143
150
 
@@ -31,7 +31,8 @@ module Bidi2pdf
31
31
 
32
32
  def on_page_loaded: (untyped _event) -> untyped
33
33
 
34
- def on_network_idle: (untyped _event) -> untyped
34
+ # A wait that timed out is its own event: a consumer reading "network_idle" can trust it.
35
+ def on_network_idle: (untyped event) -> untyped
35
36
 
36
37
  def on_console: (untyped event) -> untyped
37
38
 
@@ -26,7 +26,7 @@ module Bidi2pdf
26
26
 
27
27
  MANIFEST: { "$schema" => "https://json-schema.org/draft/2020-12/schema", "title" => "Bidi2pdf render manifest", "description" => "--manifest FILE output", "type" => "object", "required" => ::Array["schema_version" | "bidi2pdf_version" | "created_at" | "input" | "output" | "navigation" | "render"], "properties" => { "schema_version" => { "const" => 1 }, "bidi2pdf_version" => { "type" => "string" }, "created_at" => { "type" => "string", "format" => "date-time" }, "input" => { "type" => "object", "required" => ::Array["type"], "properties" => { "type" => { "type" => "string", "enum" => ::Array["url" | "html_file" | "stdin"] }, "url" => { "type" => ::Array["string" | "null"] } } }, "output" => { "type" => "object", "properties" => { "path" => { "type" => ::Array["string" | "null"] }, "pages" => { "type" => ::Array["integer" | "null"] }, "bytes" => { "type" => ::Array["integer" | "null"] }, "sha256" => { "type" => ::Array["string" | "null"] } } }, "browser" => { "type" => "object" }, "navigation" => untyped, "render" => { "type" => "object", "properties" => { "duration_ms" => { "type" => ::Array["integer" | "null"] } } }, "console" => untyped, "network_failures" => untyped, "warnings" => untyped, "headers" => { "type" => "object", "description" => "present only when custom headers were given; secret-bearing values are redacted" } } }
28
28
 
29
- EVENT: { "$schema" => "https://json-schema.org/draft/2020-12/schema", "title" => "Bidi2pdf --json-stream event", "description" => "one JSON object per stderr line while --json-stream is active. In structured mode (--json/--output -), the last line is always {event: \"result\", result: <the matching command's own schema - see \"result\"'s own oneOf>}. In human mode (--json-stream given alone), only progress events are emitted - there is no synthetic final \"result\" event, since there is no structured result being built; the human-readable log is the completion/failure signal.", "type" => "object", "required" => ::Array["schema_version" | "t_ms" | "event"], "properties" => { "schema_version" => { "const" => 1 }, "t_ms" => { "type" => "integer", "description" => "milliseconds since the render started" }, "event" => { "type" => "string", "enum" => ::Array["navigate" | "page_loaded" | "network_idle" | "console" | "print" | "screenshot" | "result"] }, "url" => { "type" => "string", "description" => "present on a navigate event" }, "level" => { "type" => "string", "description" => "present on a console event" }, "text" => { "type" => "string", "description" => "present on a console event" }, "result" => { "description" => "present only on the final \"result\" event (structured mode only); shape depends on which command produced this stream - each branch's own \"command\" value disambiguates", "oneOf" => ::Array[untyped] } } }
29
+ EVENT: { "$schema" => "https://json-schema.org/draft/2020-12/schema", "title" => "Bidi2pdf --json-stream event", "description" => "one JSON object per stderr line while --json-stream is active. In structured mode (--json/--output -), the last line is always {event: \"result\", result: <the matching command's own schema - see \"result\"'s own oneOf>}. In human mode (--json-stream given alone), only progress events are emitted - there is no synthetic final \"result\" event, since there is no structured result being built; the human-readable log is the completion/failure signal.", "type" => "object", "required" => ::Array["schema_version" | "t_ms" | "event"], "properties" => { "schema_version" => { "const" => 1 }, "t_ms" => { "type" => "integer", "description" => "milliseconds since the render started" }, "event" => { "type" => "string", "enum" => ::Array["navigate" | "page_loaded" | "network_idle" | "network_idle_timeout" | "console" | "print" | "screenshot" | "result"], "description" => "network_idle_timeout: the wait for an idle network gave up with requests still open" }, "url" => { "type" => "string", "description" => "present on a navigate event" }, "level" => { "type" => "string", "description" => "present on a console event" }, "text" => { "type" => "string", "description" => "present on a console event" }, "result" => { "description" => "present only on the final \"result\" event (structured mode only); shape depends on which command produced this stream - each branch's own \"command\" value disambiguates", "oneOf" => ::Array[untyped] } } }
30
30
 
31
31
  RECIPE_SOURCE: { "description" => "exactly one of url, file, stdin", "oneOf" => ::Array[{ "type" => "object", "required" => ::Array["url"], "properties" => { "url" => { "type" => "string" } }, "additionalProperties" => false } | { "type" => "object", "required" => ::Array["file"], "properties" => { "file" => { "type" => "string" } }, "additionalProperties" => false } | { "type" => "object", "required" => ::Array["stdin"], "properties" => { "stdin" => { "const" => true } }, "additionalProperties" => false }] }
32
32
 
@@ -72,6 +72,8 @@ module Bidi2pdf
72
72
 
73
73
  @retry_refused_sessions: untyped
74
74
 
75
+ @shutdown_timeout: untyped
76
+
75
77
  # @return [Integer] Number of Chrome slots to keep pre-warmed.
76
78
  attr_accessor size: untyped
77
79
 
@@ -120,14 +122,26 @@ module Bidi2pdf
120
122
  # pressure and try once more. Default true; false leaves a refused session to the caller.
121
123
  attr_accessor retry_refused_sessions: untyped
122
124
 
125
+ # @return [Numeric, nil] Seconds #shutdown waits, all in all, for its background threads and
126
+ # for closing the warm spares. A Chrome that hangs or ran out of memory answers no command,
127
+ # and every close would wait out its own timeout - an exiting process must not hang on that.
128
+ # What is still running then is left behind (a warning, +session_warmer.shutdown_timeout+);
129
+ # its sessions stay in the registry and a later sweep closes them. +nil+ waits for ever.
130
+ attr_accessor shutdown_timeout: untyped
131
+
123
132
  DEFAULT_MAX_IDLE_AGE: 300
124
133
 
134
+ DEFAULT_SHUTDOWN_TIMEOUT: 10
135
+
125
136
  def initialize: () -> void
126
137
 
127
138
  # The orphan age in seconds, or nil when the sweep is off (also when +:auto+ has no
128
139
  # max_idle_age to derive it from).
129
140
  def effective_orphan_age: () -> untyped
130
141
 
142
+ # nil (no limit) or a positive number of seconds - for shutdown_timeout and #shutdown's argument.
143
+ def self.validate_timeout!: (untyped name, untyped value) -> (nil | untyped)
144
+
131
145
  # @raise [ArgumentError] if a setting can't be honored - checked once, at construction.
132
146
  def validate!: () -> untyped
133
147
 
@@ -139,6 +153,8 @@ module Bidi2pdf
139
153
 
140
154
  def validate_orphan_age!: () -> (nil | untyped)
141
155
 
156
+ def validate_shutdown_timeout!: () -> untyped
157
+
142
158
  def validate_sweeper!: () -> (nil | untyped)
143
159
  end
144
160
 
@@ -204,10 +220,40 @@ module Bidi2pdf
204
220
  # finish (each of those, seeing @shutdown, retires its own result instead of stashing it - see
205
221
  # #stash_or_retire). A subsequent #with_tab still works - it just falls back to a synchronous
206
222
  # slot, since checkout never depends on a warm one being there.
207
- def shutdown: () -> untyped
223
+ #
224
+ # All of that within +timeout+ seconds together (Configuration#shutdown_timeout; nil waits for
225
+ # ever). The spares start closing first, side by side, so a hung sweep cannot use up their share
226
+ # of the deadline. Whatever is still running at the deadline is left to finish on its own, or to
227
+ # die with the process; a spare still closing loses its lease (it keeps its registry entry), so
228
+ # a sweeper can take its session even while this process lives on.
229
+ #
230
+ # @return [Hash{String => Integer}] what was still running at the deadline ("spare" => 1, ...) -
231
+ # empty when everything finished in time.
232
+ # @raise [ArgumentError] for a +timeout+ that is neither nil nor a positive number.
233
+ def shutdown: (?timeout: untyped) -> untyped
208
234
 
209
235
  private
210
236
 
237
+ def stop_accepting: () -> untyped
238
+
239
+ # The labels of whatever is still running at the deadline: reaper, sweeper, replenishments.
240
+ def stop_background: (untyped threads, untyped deadline) -> untyped
241
+
242
+ def remaining: (untyped deadline) -> untyped
243
+
244
+ # Thread#join returns nil when the time ran out first.
245
+ def unfinished: (untyped threads, untyped deadline, untyped label) -> untyped
246
+
247
+ # [slot, closing thread] pairs.
248
+ def retire_all: (untyped slots) -> untyped
249
+
250
+ # Spares still closing at the deadline, with their lease released: the heartbeat would otherwise
251
+ # keep renewing it, and no sweeper takes a leased session.
252
+ def abandoned_spares: (untyped closers, untyped deadline) -> untyped
253
+
254
+ # Only called with something pending, which needs a deadline - so +timeout+ is a number here.
255
+ def warn_abandoned: (untyped pending, untyped timeout) -> untyped
256
+
211
257
  # Fail-fast on purpose (a Chrome that can't start at boot should be loud), but not leaky: if slot
212
258
  # N fails, no instance is returned to own slots 1..N-1, so they are retired here first.
213
259
  def prewarm: () -> untyped
@@ -23,6 +23,12 @@ module Bidi2pdf
23
23
  # then sits silently in the pull for many minutes.
24
24
  DEFAULT_IMAGE: "dieters877565/chromedriver:latest"
25
25
 
26
+ # BIDI2PDF_BUILD_CHROMEDRIVER_IMAGE=true: build the image from the checkout (build_dir,
27
+ # docker_file) and run that, so a change to the image is tested before it is published -
28
+ # bidi2pdf's own CI sets it. Unset, the published image is pulled. Testcontainers' #start
29
+ # always pulls, so a locally built tag cannot go through it; #start_local_image builds instead.
30
+ def self.build_locally?: (?untyped env) -> untyped
31
+
26
32
  attr_reader docker_file: untyped
27
33
 
28
34
  attr_reader build_dir: untyped
@@ -1,3 +1,3 @@
1
1
  module Bidi2pdf
2
- VERSION: "0.1.20"
2
+ VERSION: "0.1.22"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: bidi2pdf
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.20
4
+ version: 0.1.22
5
5
  platform: ruby
6
6
  authors:
7
7
  - Dieter S.
@@ -408,8 +408,10 @@ files:
408
408
  - docker/Dockerfile.chromedriver
409
409
  - docker/Dockerfile.slim
410
410
  - docker/chrome-driver/chrome.json
411
+ - docker/chromedriver-watchdog.sh
411
412
  - docker/docker-compose.yml
412
413
  - docker/entrypoint.sh
414
+ - docker/install-hyphen-data.sh
413
415
  - docker/nginx/default.conf
414
416
  - docker/nginx/htpasswd
415
417
  - exe/bidi2pdf