bidi2pdf 0.1.19 → 0.1.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +36 -2
- data/README.md +70 -1
- data/docker/Dockerfile +13 -1
- data/docker/Dockerfile.chromedriver +20 -2
- data/docker/Dockerfile.slim +13 -1
- data/docker/chromedriver-watchdog.sh +115 -0
- data/docker/entrypoint.sh +35 -7
- data/docker/install-hyphen-data.sh +96 -0
- data/lib/bidi2pdf/chrome_sweeper/settings.rb +1 -0
- data/lib/bidi2pdf/chrome_sweeper.rb +17 -6
- data/lib/bidi2pdf/session_warmer.rb +86 -7
- data/lib/bidi2pdf/test_helpers/testcontainers/chromedriver_container.rb +12 -1
- data/lib/bidi2pdf/test_helpers/testcontainers/chromedriver_test_helper.rb +1 -1
- data/lib/bidi2pdf/version.rb +1 -1
- data/sig/bidi2pdf/chrome_sweeper/settings.rbs +2 -0
- data/sig/bidi2pdf/chrome_sweeper.rbs +10 -3
- data/sig/bidi2pdf/session_warmer.rbs +47 -1
- data/sig/bidi2pdf/test_helpers/testcontainers/chromedriver_container.rbs +10 -1
- data/sig/bidi2pdf/version.rbs +1 -1
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 93fc35bee7628afb6627782be91a2fc62caf14f8f365929c166f133d74adebfc
|
|
4
|
+
data.tar.gz: ca9381a65bd24341c2e486f34fd47f920fbf996e540c54fb421129c2f175f29e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a457600c1c6a638ee87a28d4f6221918b754a065d6d07acef9e057a4a6a3dce7c0411947cf2c340cb0a5e0ff62171bd9e934f65ca9bd15d9d514f754de11746c
|
|
7
|
+
data.tar.gz: 383a01704a6871e5e5aafcd7fdd7a97e499009fb7cba63e13adfe2f08c91ba190b78a9d2117e02fd22de1381686c61ab1c94f43fb2d6b6e92e45c9365cfb5122
|
data/CHANGELOG.md
CHANGED
|
@@ -8,10 +8,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
|
|
9
9
|
## [Unreleased]
|
|
10
10
|
|
|
11
|
-
[unreleased]: https://github.com/dieter-medium/bidi2pdf/compare/v0.1.
|
|
11
|
+
[unreleased]: https://github.com/dieter-medium/bidi2pdf/compare/v0.1.21..HEAD
|
|
12
12
|
|
|
13
13
|
<!-- generated by git-cliff end -->
|
|
14
14
|
|
|
15
|
+
## [0.1.21] - 2026-10-09
|
|
16
|
+
|
|
17
|
+
### 🐛 Fixed
|
|
18
|
+
- Define the word the hyphenation control prints
|
|
19
|
+
- Run image specs against the image from the PR
|
|
20
|
+
- Pin hyphenation data to a fixed language set
|
|
21
|
+
- Check hyphenation in the published image
|
|
22
|
+
- Move the hyphenation spec to its own PR
|
|
23
|
+
- Build chromedriver images one at a time
|
|
24
|
+
- Read watchdog settings as decimal numbers
|
|
25
|
+
- Close spares first and free their leases
|
|
26
|
+
|
|
27
|
+
### 🔄 Changed
|
|
28
|
+
- Merge pull request #159 from dieter-medium/feat/hyphenation-spec
|
|
29
|
+
- Merge pull request #158 from dieter-medium/feat/hyphenation-data
|
|
30
|
+
- Merge pull request #157 from dieter-medium/fix/chromedriver-build-race
|
|
31
|
+
- Merge pull request #156 from dieter-medium/feat/chromedriver-watchdog
|
|
32
|
+
- Merge pull request #155 from dieter-medium/feat/shutdown-deadline
|
|
33
|
+
|
|
34
|
+
### 🚀 Added
|
|
35
|
+
- Install hyphenation dictionaries in the images
|
|
36
|
+
- Restart a stuck chromedriver container
|
|
37
|
+
- Bound the warmer shutdown by a deadline
|
|
38
|
+
|
|
39
|
+
## [0.1.20] - 2026-09-30
|
|
40
|
+
|
|
41
|
+
### 🐛 Fixed
|
|
42
|
+
- Pull only the latest chromedriver image
|
|
43
|
+
|
|
44
|
+
### 🔄 Changed
|
|
45
|
+
- Merge pull request #152 from dieter-medium/fix/chromedriver-image-tag
|
|
46
|
+
|
|
15
47
|
## [0.1.19] - 2026-09-30
|
|
16
48
|
|
|
17
49
|
### 🐛 Fixed
|
|
@@ -523,7 +555,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
523
555
|
|
|
524
556
|
### 🔄 Released
|
|
525
557
|
|
|
526
|
-
- [unreleased](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.
|
|
558
|
+
- [unreleased](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.21..HEAD)
|
|
559
|
+
- [0.1.21](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.20..v0.1.21)
|
|
560
|
+
- [0.1.20](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.19..v0.1.20)
|
|
527
561
|
- [0.1.19](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.18..v0.1.19)
|
|
528
562
|
- [0.1.18](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.17..v0.1.18)
|
|
529
563
|
- [0.1.17](https://github.com/dieter-medium/bidi2pdf/compare/v0.1.16..v0.1.17)
|
data/README.md
CHANGED
|
@@ -459,6 +459,67 @@ the box in either image, no extra install step needed.
|
|
|
459
459
|
Chromium comes from Debian's packages at build time, so every build can carry a different
|
|
460
460
|
Chromium - for a byte-exact pin use the digest (`docker buildx imagetools inspect <image:tag>`).
|
|
461
461
|
|
|
462
|
+
### Hyphenation
|
|
463
|
+
|
|
464
|
+
The images (`chromedriver`, `bidi2pdf` and its slim variant) ship Chromium's hyphenation dictionaries
|
|
465
|
+
in `/usr/lib/chromium/hyphen-data/`, so CSS `hyphens: auto` breaks long words in the page's language
|
|
466
|
+
(`lang="de"` on the element or the document) - for exactly these languages, by the `lang` tag Blink
|
|
467
|
+
maps to each dictionary (case does not matter; Blink drops trailing subtags until one matches):
|
|
468
|
+
|
|
469
|
+
| `lang` | Dictionary |
|
|
470
|
+
|---|---|
|
|
471
|
+
| `de`, `de-DE`, `de-AT`, `de-CH` (and `de-DE-1901` - the region wins) | German, 1996 spelling (`de-1996`) |
|
|
472
|
+
| `de-1901` | German, traditional spelling |
|
|
473
|
+
| `de-CH-1901`, `de-LI-1901` | Swiss German, traditional spelling |
|
|
474
|
+
| `en`, `en-US` | US English |
|
|
475
|
+
| `en-GB`, `en-AU`, `en-IE`, ... (every other `en-*`) | British English |
|
|
476
|
+
| `fr`, `es`, `it`, `nl`, `pt` (with any region, e.g. `pt-BR`) | French, Spanish, Italian, Dutch, Portuguese |
|
|
477
|
+
|
|
478
|
+
The hyphenation spec prints one word in each of these, against a control without hyphenation.
|
|
479
|
+
Debian's Chromium has none, and the component it would download needs a profile that outlives the
|
|
480
|
+
session. `docker/install-hyphen-data.sh` fetches them at build time from Chromium's source, pinned by
|
|
481
|
+
the `HYPHEN_DATA_REF` build argument - deliberately independent of the Debian Chromium the image
|
|
482
|
+
installs, since the pattern data and its format change far more rarely; the hyphenation spec catches a
|
|
483
|
+
Chromium that stops reading them - with their licence notice (`LICENSE` next to them). Do not pass `--disable-component-update` in
|
|
484
|
+
`chrome_args`: Chromium then skips the dictionaries.
|
|
485
|
+
|
|
486
|
+
### ChromeDriver health check and watchdog
|
|
487
|
+
|
|
488
|
+
The image has a Docker `HEALTHCHECK` (chromedriver's `/status`), so `docker ps` shows a
|
|
489
|
+
chromedriver that stopped answering as `unhealthy`. It does not see a Chrome that is wedged behind
|
|
490
|
+
a chromedriver that still answers - memory thrashing shows only to the watchdog. And Docker does
|
|
491
|
+
not restart a container for being unhealthy - only when its main process exits.
|
|
492
|
+
|
|
493
|
+
That is what the opt-in watchdog does. With `WATCHDOG_ENABLED=true` it checks every
|
|
494
|
+
`WATCHDOG_INTERVAL` seconds and, after `WATCHDOG_FAILURES` failed checks in a row, stops
|
|
495
|
+
chromedriver; the container exits non-zero and its restart policy starts a fresh one - any policy
|
|
496
|
+
but `no` (`--restart unless-stopped` or `on-failure`, `restart:` in Compose; Kamal sets
|
|
497
|
+
`unless-stopped` for accessories). Without one, a watchdog restart just stops the container. A
|
|
498
|
+
check fails when chromedriver does not answer `/status`, or when the
|
|
499
|
+
container's memory thrashes: Linux's pressure stall information (`memory.pressure`, "full") shows
|
|
500
|
+
every task stalled on memory for at least `WATCHDOG_MEMORY_PRESSURE` percent of the last 10 s - a
|
|
501
|
+
Chrome swapping at its memory limit, which answers no command but never crashes.
|
|
502
|
+
|
|
503
|
+
Renders still running on that Chrome fail when it restarts; a client with a sweeper retries them.
|
|
504
|
+
|
|
505
|
+
| Variable | Default | Description |
|
|
506
|
+
|----------------------------|----------------------------------|--------------------------------------------------------------|
|
|
507
|
+
| `WATCHDOG_ENABLED` | `false` | `true` starts the watchdog. |
|
|
508
|
+
| `WATCHDOG_INTERVAL` | `15` | Seconds between checks. |
|
|
509
|
+
| `WATCHDOG_FAILURES` | `3` | Failed checks in a row before the restart. |
|
|
510
|
+
| `WATCHDOG_STATUS_TIMEOUT` | `5` | Seconds `/status` may take. |
|
|
511
|
+
| `WATCHDOG_MEMORY_PRESSURE` | `50` | "full avg10" percent that fails a check; `0` turns it off. |
|
|
512
|
+
| `WATCHDOG_STOP_GRACE` | `10` | Seconds between stopping chromedriver (TERM) and killing it. |
|
|
513
|
+
| `WATCHDOG_PRESSURE_FILE` | `/sys/fs/cgroup/memory.pressure` | The PSI file; without one (cgroup v1, PSI off) only `/status` counts. |
|
|
514
|
+
|
|
515
|
+
All values are whole numbers (seconds, percent). A malformed value stops the watchdog from
|
|
516
|
+
starting - chromedriver keeps running unwatched, and the container log says why.
|
|
517
|
+
|
|
518
|
+
Picking `WATCHDOG_MEMORY_PRESSURE`: "full" counts only the time *every* task waited on memory, so it
|
|
519
|
+
stays near 0 while renders merely run; 50 % of the last 10 s is a container that is stuck, not busy.
|
|
520
|
+
Before lowering it, watch `/sys/fs/cgroup/memory.pressure` inside the container under your usual
|
|
521
|
+
render load and stay well above what you see there.
|
|
522
|
+
|
|
462
523
|
### Docker Compose
|
|
463
524
|
|
|
464
525
|
```bash
|
|
@@ -574,6 +635,7 @@ Bidi2pdf::SessionWarmer.shutdown
|
|
|
574
635
|
| `registry_dir` | `Dir.tmpdir` | Where the session registry file lives - every process that should clean up after the others must share it. |
|
|
575
636
|
| `sweeper` | `nil` | Remote only: settings for a [`ChromeSweeper`](#leaked-chrome-sessions-bidi2pdfchromesweeper) on the same chromedriver, e.g. `{ scope: :all, max_sessions: :auto, pids_limit: 1024, interval: 60 }`. The warmer's own sessions are never touched; `Bidi2pdf::SessionWarmer.sweep!` sweeps on demand. |
|
|
576
637
|
| `retry_refused_sessions` | `true` | With a `sweeper`: when chromedriver refuses a new session, sweep under pressure and try once more. `false` leaves it to the caller. |
|
|
638
|
+
| `shutdown_timeout` | `10` | Seconds `shutdown` waits, all in all, for its background threads and for closing the warm sessions. `nil` waits for ever. |
|
|
577
639
|
|
|
578
640
|
#### Leftover sessions on a shared chromedriver
|
|
579
641
|
|
|
@@ -595,6 +657,13 @@ With a `sweeper` configured, the warmer also sweeps in the background every `int
|
|
|
595
657
|
and when chromedriver refuses a new session ("session not created" - typically a container out of
|
|
596
658
|
room for another Chrome) it sweeps once and tries once more.
|
|
597
659
|
|
|
660
|
+
`shutdown` is bounded by `shutdown_timeout`. A Chrome that hangs or ran out of memory answers no
|
|
661
|
+
command, and closing each of its sessions would wait out every command timeout in turn - long
|
|
662
|
+
enough that a stopping server looks hung. At the deadline the warmer stops waiting, logs what is
|
|
663
|
+
still running and instruments `session_warmer.shutdown_timeout.bidi2pdf` (`timeout`, `pending`);
|
|
664
|
+
`shutdown` returns the same tally, e.g. `{ "spare" => 1 }`, and `{}` when everything finished.
|
|
665
|
+
Sessions it could not close stay in the registry, so a later sweep closes them.
|
|
666
|
+
|
|
598
667
|
> **Security note:** an idle warm session is an open, unauthenticated automation endpoint
|
|
599
668
|
> (chromedriver's port, Chrome's debugging port - loopback only for a local chromedriver) for as
|
|
600
669
|
> long as it waits. `max_idle_age` bounds that window; keep it set unless the process runs somewhere
|
|
@@ -663,7 +732,7 @@ session is refused.
|
|
|
663
732
|
| `pids_limit` | `nil` | The chromedriver container's pids limit (Docker counts threads). `:auto` without it means no limit. |
|
|
664
733
|
| `pids_budget` | `0.8` | Share of `pids_limit` the Chrome sessions may use. |
|
|
665
734
|
| `threads_per_session` | `110` | Threads one Chrome session uses. |
|
|
666
|
-
| `interval` | `nil` | Seconds between background sweeps (`start`/`stop`).
|
|
735
|
+
| `interval` | `nil` | Seconds between background sweeps (`start`/`stop`). `stop(timeout: 10)` waits that long for a sweep in progress, then stops waiting and returns `false` (`nil` waits for ever). |
|
|
667
736
|
| `dry_run` | `false` | Report what would be closed, close nothing. |
|
|
668
737
|
| `registry_dir` | `Dir.tmpdir`| The registry to read and update - same meaning as the warmer's setting. |
|
|
669
738
|
| `own_sessions` | `-> { [] }` | A callable returning the caller's live session ids; they are never touched. |
|
data/docker/Dockerfile
CHANGED
|
@@ -13,11 +13,23 @@ RUN apt-get update && apt-get upgrade -y &&\
|
|
|
13
13
|
libatk-bridge2.0-0 \
|
|
14
14
|
libgtk-3-0 \
|
|
15
15
|
libdrm2 \
|
|
16
|
-
curl \
|
|
16
|
+
curl ca-certificates \
|
|
17
17
|
unzip \
|
|
18
18
|
xvfb \
|
|
19
19
|
&& rm -rf /var/lib/apt/lists/*
|
|
20
20
|
|
|
21
|
+
# Chromium's hyphenation dictionaries, so that `hyphens: auto` works in the PDFs. Debian's chromium
|
|
22
|
+
# ships none and Chrome cannot download them here - see docker/install-hyphen-data.sh for the whole
|
|
23
|
+
# story. They go next to the chromium binary, where the component installer looks for a bundled copy.
|
|
24
|
+
# The patterns are data files and intentionally do not track the Debian Chromium installed above (the
|
|
25
|
+
# minikin .hyb format is unchanged since 2016, the files since 2023): update this ref when the pattern
|
|
26
|
+
# data itself changes. The hyphenation integration spec, run against the image's own Chromium, catches
|
|
27
|
+
# a Chromium that stops reading them.
|
|
28
|
+
ARG HYPHEN_DATA_REF=refs/tags/154.0.8037.92
|
|
29
|
+
COPY ./docker/install-hyphen-data.sh /usr/local/bin/install-hyphen-data.sh
|
|
30
|
+
RUN chmod +x /usr/local/bin/install-hyphen-data.sh && \
|
|
31
|
+
/usr/local/bin/install-hyphen-data.sh "${HYPHEN_DATA_REF}" /usr/lib/chromium/hyphen-data
|
|
32
|
+
|
|
21
33
|
# Create a non-root user
|
|
22
34
|
RUN groupadd -r appuser && useradd -r -g appuser -m -d /home/appuser appuser
|
|
23
35
|
|
|
@@ -18,7 +18,7 @@ RUN echo "deb http://deb.debian.org/debian bookworm contrib non-free" > /etc/apt
|
|
|
18
18
|
libatk-bridge2.0-0 \
|
|
19
19
|
libgtk-3-0 \
|
|
20
20
|
libdrm2 \
|
|
21
|
-
curl \
|
|
21
|
+
curl ca-certificates \
|
|
22
22
|
unzip \
|
|
23
23
|
xvfb \
|
|
24
24
|
x11vnc \
|
|
@@ -34,11 +34,24 @@ RUN echo "deb http://deb.debian.org/debian bookworm contrib non-free" > /etc/apt
|
|
|
34
34
|
&& locale-gen en_US.UTF-8 \
|
|
35
35
|
&& rm -rf /var/lib/apt/lists/*
|
|
36
36
|
|
|
37
|
+
# Chromium's hyphenation dictionaries, so that `hyphens: auto` works in the PDFs. Debian's chromium
|
|
38
|
+
# ships none and Chrome cannot download them here - see docker/install-hyphen-data.sh for the whole
|
|
39
|
+
# story. They go next to the chromium binary, where the component installer looks for a bundled copy.
|
|
40
|
+
# The patterns are data files and intentionally do not track the Debian Chromium installed above (the
|
|
41
|
+
# minikin .hyb format is unchanged since 2016, the files since 2023): update this ref when the pattern
|
|
42
|
+
# data itself changes. The hyphenation integration spec, run against the image's own Chromium, catches
|
|
43
|
+
# a Chromium that stops reading them.
|
|
44
|
+
ARG HYPHEN_DATA_REF=refs/tags/154.0.8037.92
|
|
45
|
+
COPY ./docker/install-hyphen-data.sh /usr/local/bin/install-hyphen-data.sh
|
|
46
|
+
RUN chmod +x /usr/local/bin/install-hyphen-data.sh && \
|
|
47
|
+
/usr/local/bin/install-hyphen-data.sh "${HYPHEN_DATA_REF}" /usr/lib/chromium/hyphen-data
|
|
48
|
+
|
|
37
49
|
# Create a non-root user
|
|
38
50
|
RUN groupadd -r appuser && useradd -r -g appuser -G audio,video -m -d /home/appuser appuser
|
|
39
51
|
|
|
40
52
|
COPY ./docker/entrypoint.sh /usr/local/bin/entrypoint.sh
|
|
41
|
-
|
|
53
|
+
COPY ./docker/chromedriver-watchdog.sh /usr/local/bin/chromedriver-watchdog.sh
|
|
54
|
+
RUN chmod +x /usr/local/bin/entrypoint.sh /usr/local/bin/chromedriver-watchdog.sh
|
|
42
55
|
|
|
43
56
|
# ARM compatibility workaround:
|
|
44
57
|
# On ARM architectures (such as Apple Silicon), downloading chromedriver via automated scripts may fail or cause ELF binary errors,
|
|
@@ -69,4 +82,9 @@ EXPOSE ${CHROMEDRIVER_PORT}
|
|
|
69
82
|
# VNC
|
|
70
83
|
EXPOSE 5900
|
|
71
84
|
|
|
85
|
+
# Marks the container "unhealthy" in docker ps when chromedriver stops answering. Docker does not
|
|
86
|
+
# restart it for that - WATCHDOG_ENABLED=true does (chromedriver-watchdog.sh).
|
|
87
|
+
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
|
88
|
+
CMD curl -fs -m 4 -o /dev/null "http://127.0.0.1:${CHROMEDRIVER_PORT}/status" || exit 1
|
|
89
|
+
|
|
72
90
|
CMD ["/usr/local/bin/entrypoint.sh"]
|
data/docker/Dockerfile.slim
CHANGED
|
@@ -50,13 +50,25 @@ RUN apt-get update && apt-get upgrade -y &&\
|
|
|
50
50
|
libatk-bridge2.0-0 \
|
|
51
51
|
libgtk-3-0 \
|
|
52
52
|
libdrm2 \
|
|
53
|
-
curl \
|
|
53
|
+
curl ca-certificates \
|
|
54
54
|
unzip \
|
|
55
55
|
xvfb \
|
|
56
56
|
&& rm -rf /var/lib/apt/lists/*
|
|
57
57
|
|
|
58
58
|
COPY --from=builder /usr/local/bundle /usr/local/bundle
|
|
59
59
|
|
|
60
|
+
# Chromium's hyphenation dictionaries, so that `hyphens: auto` works in the PDFs. Debian's chromium
|
|
61
|
+
# ships none and Chrome cannot download them here - see docker/install-hyphen-data.sh for the whole
|
|
62
|
+
# story. They go next to the chromium binary, where the component installer looks for a bundled copy.
|
|
63
|
+
# The patterns are data files and intentionally do not track the Debian Chromium installed above (the
|
|
64
|
+
# minikin .hyb format is unchanged since 2016, the files since 2023): update this ref when the pattern
|
|
65
|
+
# data itself changes. The hyphenation integration spec, run against the image's own Chromium, catches
|
|
66
|
+
# a Chromium that stops reading them.
|
|
67
|
+
ARG HYPHEN_DATA_REF=refs/tags/154.0.8037.92
|
|
68
|
+
COPY ./docker/install-hyphen-data.sh /usr/local/bin/install-hyphen-data.sh
|
|
69
|
+
RUN chmod +x /usr/local/bin/install-hyphen-data.sh && \
|
|
70
|
+
/usr/local/bin/install-hyphen-data.sh "${HYPHEN_DATA_REF}" /usr/lib/chromium/hyphen-data
|
|
71
|
+
|
|
60
72
|
# Create a non-root user
|
|
61
73
|
RUN groupadd -r appuser && useradd -r -g appuser -m -d /home/appuser appuser
|
|
62
74
|
|
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
#!/bin/bash
|
|
2
|
+
# Restarts the chromedriver container when Chrome is in a state it does not recover from by itself:
|
|
3
|
+
# chromedriver no longer answers /status, or the container's memory is thrashing (PSI "full" - every
|
|
4
|
+
# task stalled on memory for that share of the last 10 s). It stops chromedriver, entrypoint.sh
|
|
5
|
+
# exits, and the container's restart policy (unless-stopped, also Kamal's for accessories) starts a
|
|
6
|
+
# fresh one. Docker itself never restarts a merely "unhealthy" container. Renders still running on
|
|
7
|
+
# that Chrome fail - the client's retry and sweeper handle that.
|
|
8
|
+
#
|
|
9
|
+
# Opt-in: entrypoint.sh starts it when WATCHDOG_ENABLED=true.
|
|
10
|
+
# Usage: chromedriver-watchdog.sh <chromedriver pid> [marker file]
|
|
11
|
+
# Before a restart it creates the marker file, so entrypoint.sh can exit non-zero and a restart
|
|
12
|
+
# policy of on-failure restarts the container too.
|
|
13
|
+
#
|
|
14
|
+
# WATCHDOG_INTERVAL seconds between checks (default 15)
|
|
15
|
+
# WATCHDOG_FAILURES failed checks in a row before restart (default 3)
|
|
16
|
+
# WATCHDOG_STATUS_TIMEOUT seconds /status may take (default 5)
|
|
17
|
+
# WATCHDOG_MEMORY_PRESSURE PSI full avg10 percent that fails a check, 0 = off (default 50)
|
|
18
|
+
# WATCHDOG_STOP_GRACE seconds between TERM and KILL (default 10)
|
|
19
|
+
# WATCHDOG_PRESSURE_FILE the cgroup's PSI file (default /sys/fs/cgroup/memory.pressure)
|
|
20
|
+
set -euo pipefail
|
|
21
|
+
|
|
22
|
+
pid="${1:-}"
|
|
23
|
+
marker="${2:-}"
|
|
24
|
+
port="${CHROMEDRIVER_PORT:-3000}"
|
|
25
|
+
interval="${WATCHDOG_INTERVAL:-15}"
|
|
26
|
+
failures="${WATCHDOG_FAILURES:-3}"
|
|
27
|
+
status_timeout="${WATCHDOG_STATUS_TIMEOUT:-5}"
|
|
28
|
+
pressure_limit="${WATCHDOG_MEMORY_PRESSURE:-50}"
|
|
29
|
+
stop_grace="${WATCHDOG_STOP_GRACE:-10}"
|
|
30
|
+
pressure_file="${WATCHDOG_PRESSURE_FILE:-/sys/fs/cgroup/memory.pressure}"
|
|
31
|
+
reason=""
|
|
32
|
+
|
|
33
|
+
log() {
|
|
34
|
+
printf 'chromedriver-watchdog: %s\n' "$*" >&2
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
# Prints the value as a plain decimal - "050" is 50, never bash's octal 40 - or fails closed on a
|
|
38
|
+
# malformed setting: the watchdog does not start (chromedriver keeps running).
|
|
39
|
+
decimal() {
|
|
40
|
+
local name="$1" value="$2" min="$3"
|
|
41
|
+
case "$value" in
|
|
42
|
+
'' | *[!0-9]*) ;;
|
|
43
|
+
*)
|
|
44
|
+
if ((10#$value >= min)); then
|
|
45
|
+
printf '%d\n' "$((10#$value))"
|
|
46
|
+
return
|
|
47
|
+
fi
|
|
48
|
+
;;
|
|
49
|
+
esac
|
|
50
|
+
log "error: ${name} must be an integer >= ${min}, got '${value}' - not watching"
|
|
51
|
+
exit 1
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
pid="$(decimal "chromedriver pid" "$pid" 1)"
|
|
55
|
+
port="$(decimal CHROMEDRIVER_PORT "$port" 1)"
|
|
56
|
+
interval="$(decimal WATCHDOG_INTERVAL "$interval" 1)"
|
|
57
|
+
failures="$(decimal WATCHDOG_FAILURES "$failures" 1)"
|
|
58
|
+
status_timeout="$(decimal WATCHDOG_STATUS_TIMEOUT "$status_timeout" 1)"
|
|
59
|
+
pressure_limit="$(decimal WATCHDOG_MEMORY_PRESSURE "$pressure_limit" 0)"
|
|
60
|
+
stop_grace="$(decimal WATCHDOG_STOP_GRACE "$stop_grace" 1)"
|
|
61
|
+
|
|
62
|
+
# Whole percent of the last 10 s in which every task of this cgroup stalled on memory; 0 when the
|
|
63
|
+
# kernel offers no PSI file (cgroup v1, PSI disabled). Truncated, not rounded: 49.9 is 49, so a
|
|
64
|
+
# limit of 50 means at least 50.0.
|
|
65
|
+
memory_pressure() {
|
|
66
|
+
if [[ ! -r "$pressure_file" ]]; then
|
|
67
|
+
printf '0\n'
|
|
68
|
+
return
|
|
69
|
+
fi
|
|
70
|
+
awk '$1 == "full" { split($2, kv, "="); printf "%d\n", kv[2]; found = 1 } END { if (!found) print 0 }' "$pressure_file"
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
check() {
|
|
74
|
+
local pressure
|
|
75
|
+
if ! curl -fs -m "$status_timeout" -o /dev/null "http://127.0.0.1:${port}/status"; then
|
|
76
|
+
reason="chromedriver did not answer /status within ${status_timeout}s"
|
|
77
|
+
return 1
|
|
78
|
+
fi
|
|
79
|
+
if ((pressure_limit > 0)); then
|
|
80
|
+
pressure="$(memory_pressure)"
|
|
81
|
+
if ((pressure >= pressure_limit)); then
|
|
82
|
+
reason="memory pressure ${pressure}% (full avg10), limit ${pressure_limit}%"
|
|
83
|
+
return 1
|
|
84
|
+
fi
|
|
85
|
+
fi
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
restart() {
|
|
89
|
+
log "restarting: stopping chromedriver (pid ${pid}) so the container exits and its restart policy starts a fresh one"
|
|
90
|
+
if [[ -n "$marker" ]] && ! : >"$marker"; then
|
|
91
|
+
log "could not write ${marker} - the container may exit 0"
|
|
92
|
+
fi
|
|
93
|
+
kill -TERM "$pid" 2>/dev/null || true
|
|
94
|
+
for ((waited = 0; waited < stop_grace; waited++)); do
|
|
95
|
+
kill -0 "$pid" 2>/dev/null || exit 0
|
|
96
|
+
sleep 1
|
|
97
|
+
done
|
|
98
|
+
kill -KILL "$pid" 2>/dev/null || true
|
|
99
|
+
exit 0
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
log "watching chromedriver (pid ${pid}) every ${interval}s, restart after ${failures} failed checks"
|
|
103
|
+
failed=0
|
|
104
|
+
while kill -0 "$pid" 2>/dev/null; do
|
|
105
|
+
sleep "$interval"
|
|
106
|
+
if check; then
|
|
107
|
+
failed=0
|
|
108
|
+
continue
|
|
109
|
+
fi
|
|
110
|
+
failed=$((failed + 1))
|
|
111
|
+
log "check ${failed}/${failures} failed: ${reason}"
|
|
112
|
+
if ((failed >= failures)); then
|
|
113
|
+
restart
|
|
114
|
+
fi
|
|
115
|
+
done
|
data/docker/entrypoint.sh
CHANGED
|
@@ -17,7 +17,7 @@ if [ "$ENABLE_XVFB" = "true" ]; then
|
|
|
17
17
|
|
|
18
18
|
xauth generate :99 . trusted
|
|
19
19
|
|
|
20
|
-
umask $old_umask
|
|
20
|
+
umask "$old_umask"
|
|
21
21
|
|
|
22
22
|
|
|
23
23
|
|
|
@@ -38,17 +38,45 @@ if [ "$ENABLE_VNC" = "true" ]; then
|
|
|
38
38
|
old_umask=$(umask)
|
|
39
39
|
umask 077
|
|
40
40
|
mkdir -p /home/appuser/.vnc
|
|
41
|
-
x11vnc -storepasswd $VNC_PASS /home/appuser/.vnc/passwd
|
|
42
|
-
umask $old_umask
|
|
41
|
+
x11vnc -storepasswd "$VNC_PASS" /home/appuser/.vnc/passwd
|
|
42
|
+
umask "$old_umask"
|
|
43
43
|
x11vnc -display WAIT:99 -xkb -noxrecord -noxfixes -noxdamage -forever -shared -noshm -usepw -rfbauth /home/appuser/.vnc/passwd &
|
|
44
44
|
fi
|
|
45
45
|
|
|
46
46
|
# DISPLAY=:99 /home/appuser/.webdrivers/chromedriver --port=33259 --whitelisted-ips="" --allowed-origins="*" --disable-dev-shm-usage --disable-gpu --verbose
|
|
47
|
-
/home/appuser/.webdrivers/chromedriver --port
|
|
47
|
+
/home/appuser/.webdrivers/chromedriver --port="${CHROMEDRIVER_PORT}" \
|
|
48
48
|
--allowed-ips="" \
|
|
49
49
|
--allowed-origins="*" \
|
|
50
50
|
--disable-dev-shm-usage \
|
|
51
51
|
--disable-gpu \
|
|
52
|
-
--user-data-dir
|
|
53
|
-
--log-level
|
|
54
|
-
--readable-timestamp
|
|
52
|
+
--user-data-dir="${USER_DATA_DIR}" \
|
|
53
|
+
--log-level="${CHROMEDRIVER_LOG_LEVEL:-INFO}" \
|
|
54
|
+
--readable-timestamp &
|
|
55
|
+
chromedriver_pid=$!
|
|
56
|
+
|
|
57
|
+
# docker stop sends TERM to this script (PID 1): pass it on so chromedriver ends its sessions.
|
|
58
|
+
trap 'kill -TERM "$chromedriver_pid" 2>/dev/null' TERM INT
|
|
59
|
+
|
|
60
|
+
# Opt-in watchdog (see chromedriver-watchdog.sh): stops chromedriver when it no longer answers or
|
|
61
|
+
# memory thrashes, so this script exits and the restart policy starts a fresh container. It leaves
|
|
62
|
+
# a marker first, in a private temp dir, so the exit below is non-zero for any restart policy.
|
|
63
|
+
watchdog_marker=""
|
|
64
|
+
if [ "${WATCHDOG_ENABLED:-false}" = "true" ]; then
|
|
65
|
+
if watchdog_dir="$(mktemp -d)"; then
|
|
66
|
+
watchdog_marker="${watchdog_dir}/restarted"
|
|
67
|
+
fi
|
|
68
|
+
/usr/local/bin/chromedriver-watchdog.sh "$chromedriver_pid" "$watchdog_marker" &
|
|
69
|
+
fi
|
|
70
|
+
|
|
71
|
+
# A trapped signal interrupts wait before chromedriver has ended: wait again until it has.
|
|
72
|
+
status=0
|
|
73
|
+
wait "$chromedriver_pid" || status=$?
|
|
74
|
+
while kill -0 "$chromedriver_pid" 2>/dev/null; do
|
|
75
|
+
status=0
|
|
76
|
+
wait "$chromedriver_pid" || status=$?
|
|
77
|
+
done
|
|
78
|
+
|
|
79
|
+
if [ -n "$watchdog_marker" ] && [ -e "$watchdog_marker" ] && [ "$status" -eq 0 ]; then
|
|
80
|
+
status=1
|
|
81
|
+
fi
|
|
82
|
+
exit "$status"
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
#!/bin/bash
|
|
2
|
+
# Installs Chromium's hyphenation dictionaries so that `hyphens: auto` works in the PDFs.
|
|
3
|
+
#
|
|
4
|
+
# Why this exists: Chromium on Linux does not ship its hyphenation dictionaries in the binary. They
|
|
5
|
+
# come as a "Hyphenation" component which the component updater downloads at runtime into
|
|
6
|
+
# <user-data-dir>/hyphen-data/<version>/. In our containers that never sticks: chromedriver makes a
|
|
7
|
+
# fresh, throw-away user-data-dir for every session (a download would be gone with it), a sandboxed
|
|
8
|
+
# remote-chrome has no internet access to download from, and Debian's chromium package (chromium,
|
|
9
|
+
# chromium-common) ships no hyphen-data at all. So `hyphens: auto` silently renders as
|
|
10
|
+
# `hyphens: manual` - no automatic word breaks, however `lang` is set.
|
|
11
|
+
#
|
|
12
|
+
# How it works: before looking in the user-data-dir, Chromium's component installer looks for a
|
|
13
|
+
# "preinstalled" copy of every component next to its own binary (ComponentInstaller::StartRegistration ->
|
|
14
|
+
# FindPreinstallation, rooted at chrome::DIR_COMPONENTS = base::DIR_ASSETS = the directory of the chromium
|
|
15
|
+
# executable, /usr/lib/chromium on Debian). It expects <root>/hyphen-data/manifest.json with a valid
|
|
16
|
+
# "version" plus the hyph-<locale>.hyb files. That is exactly how Chrome for Testing bundles them
|
|
17
|
+
# (chrome/BUILD.gn -> third_party/hyphenation-patterns:bundle_hyphen_data), with the same bogus version
|
|
18
|
+
# "1.0.0.0" that is used below. The dictionaries themselves are the AOSP minikin .hyb files that live in
|
|
19
|
+
# chromium/src at third_party/hyphenation-patterns/hyb and are fetched from there, pinned to a Chromium tag
|
|
20
|
+
# (unchanged since 2023 - the de/en files of 151 and 154 are byte-identical). The pin intentionally does
|
|
21
|
+
# not follow the Debian Chromium the image installs: the patterns are data in a format unchanged since
|
|
22
|
+
# 2016, updated when the pattern data changes; the hyphenation integration spec, run against the
|
|
23
|
+
# image's own Chromium, catches a Chromium that stops reading them.
|
|
24
|
+
#
|
|
25
|
+
# The component is only registered by the full browser (chrome_browser_main.cc), never by the old
|
|
26
|
+
# headless shell - fine for Debian's chromium >= 132, where --headless is the new headless mode - and
|
|
27
|
+
# not when --disable-component-update is passed, which neither chromedriver nor bidi2pdf does. Never add it.
|
|
28
|
+
#
|
|
29
|
+
# Usage: install-hyphen-data.sh [chromium-ref] [target-dir]
|
|
30
|
+
set -euo pipefail
|
|
31
|
+
|
|
32
|
+
ref="${1:-${HYPHEN_DATA_REF:-refs/tags/154.0.8037.92}}"
|
|
33
|
+
target="${2:-/usr/lib/chromium/hyphen-data}"
|
|
34
|
+
source_dir="third_party/hyphenation-patterns"
|
|
35
|
+
archive_url="https://chromium.googlesource.com/chromium/src/+archive/${ref}/${source_dir}.tar.gz"
|
|
36
|
+
|
|
37
|
+
# The image's language contract: exactly these dictionaries are installed, and each must be in the
|
|
38
|
+
# download or the build fails - a renamed or dropped dictionary shows up at build time, not as a silent
|
|
39
|
+
# fallback to no hyphenation, and the image does not grow when Chromium adds languages. The archive
|
|
40
|
+
# carries ~50; add a language here (and to README "Hyphenation") to support it. Inside Blink "de" maps
|
|
41
|
+
# to de-1996, "en" tries en-gb, then en-us
|
|
42
|
+
# (third_party/blink/renderer/platform/text/hyphenation/hyphenation_minikin.cc, MapLocale).
|
|
43
|
+
dictionaries="hyph-de-1996.hyb hyph-de-1901.hyb hyph-de-ch-1901.hyb hyph-en-us.hyb hyph-en-gb.hyb hyph-fr.hyb hyph-es.hyb hyph-it.hyb hyph-nl.hyb hyph-pt.hyb"
|
|
44
|
+
|
|
45
|
+
workdir="$(mktemp -d)"
|
|
46
|
+
trap 'rm -rf "$workdir"' EXIT
|
|
47
|
+
mkdir -p "${workdir}/src/hyb"
|
|
48
|
+
|
|
49
|
+
# Gitiles exports a subtree as one tarball. If that is unreachable, fall back to the GitHub mirror of
|
|
50
|
+
# chromium/src (tags are mirrored), one file at a time, with the file list taken from the same BUILD.gn
|
|
51
|
+
# that Chrome for Testing bundles from.
|
|
52
|
+
fetch_from_gitiles() {
|
|
53
|
+
echo "Fetching Chromium hyphenation dictionaries from ${archive_url}"
|
|
54
|
+
curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/hyb.tar.gz" "${archive_url}" &&
|
|
55
|
+
tar -xzf "${workdir}/hyb.tar.gz" -C "${workdir}/src"
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
fetch_from_github() {
|
|
59
|
+
local raw="https://raw.githubusercontent.com/chromium/chromium/${ref#refs/tags/}/${source_dir}"
|
|
60
|
+
echo "Fetching Chromium hyphenation dictionaries from ${raw}"
|
|
61
|
+
curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/BUILD.gn" "${raw}/BUILD.gn"
|
|
62
|
+
curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/src/LICENSE" "${raw}/LICENSE"
|
|
63
|
+
grep -o '"hyb/hyph-[^"]*\.hyb"' "${workdir}/BUILD.gn" | tr -d '"' | sort -u | while read -r file; do
|
|
64
|
+
curl -fsSL --retry 3 --retry-delay 2 -o "${workdir}/src/${file}" "${raw}/${file}"
|
|
65
|
+
done
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
fetch_from_gitiles || { echo "gitiles download failed, trying the GitHub mirror" >&2; fetch_from_github; }
|
|
69
|
+
|
|
70
|
+
mkdir -p "${target}"
|
|
71
|
+
for file in ${dictionaries}; do
|
|
72
|
+
if [ ! -s "${workdir}/src/hyb/${file}" ]; then
|
|
73
|
+
echo "hyphenation dictionary ${file} missing from ${archive_url}" >&2
|
|
74
|
+
exit 1
|
|
75
|
+
fi
|
|
76
|
+
install -m 0644 "${workdir}/src/hyb/${file}" "${target}/${file}"
|
|
77
|
+
done
|
|
78
|
+
# The patterns are TeX hyphenation patterns under a mix of licenses; keep their notice with the data
|
|
79
|
+
# (Chromium's file covers every language, these included).
|
|
80
|
+
install -m 0644 "${workdir}/src/LICENSE" "${target}/LICENSE"
|
|
81
|
+
|
|
82
|
+
# Chromium's component installer only accepts the directory with a manifest carrying a valid version.
|
|
83
|
+
# 1.0.0.0 is what Chrome for Testing uses for its bundled copy: lower than anything the component
|
|
84
|
+
# updater would ever serve, and not all zeros (that means "no component").
|
|
85
|
+
cat > "${target}/manifest.json" <<'EOF'
|
|
86
|
+
{
|
|
87
|
+
"manifest_version": 2,
|
|
88
|
+
"name": "hyphens-data",
|
|
89
|
+
"version": "1.0.0.0"
|
|
90
|
+
}
|
|
91
|
+
EOF
|
|
92
|
+
chmod 0644 "${target}/manifest.json"
|
|
93
|
+
chmod 0755 "${target}"
|
|
94
|
+
|
|
95
|
+
dictionaries=("${target}"/*.hyb)
|
|
96
|
+
echo "Installed ${#dictionaries[@]} hyphenation dictionaries to ${target}"
|
|
@@ -9,6 +9,7 @@ module Bidi2pdf
|
|
|
9
9
|
DEFAULT_PIDS_BUDGET = 0.8
|
|
10
10
|
DEFAULT_THREADS_PER_SESSION = 110
|
|
11
11
|
DEFAULT_LEASE_TTL = SessionRegistry::DEFAULT_LEASE_TTL
|
|
12
|
+
DEFAULT_STOP_TIMEOUT = 10
|
|
12
13
|
|
|
13
14
|
# What a ChromeSweeper closes and when - validated on construction; an unknown setting raises
|
|
14
15
|
# ArgumentError.
|
|
@@ -167,21 +167,32 @@ module Bidi2pdf
|
|
|
167
167
|
end
|
|
168
168
|
end
|
|
169
169
|
|
|
170
|
-
# Starts the periodic sweep thread (needs an +interval+). Idempotent.
|
|
170
|
+
# Starts the periodic sweep thread (needs an +interval+). Idempotent. Each thread has its own
|
|
171
|
+
# wakeup queue, so one that #stop stopped waiting for can never take a later thread's :stop.
|
|
172
|
+
# After such a #stop, a new thread can sweep while the abandoned sweep is still running: the
|
|
173
|
+
# lock file lets only one of them sweep at a time, the other's sweep is skipped.
|
|
171
174
|
def start
|
|
172
175
|
raise Bidi2pdf::InvalidConfigError, "chrome_sweeper: start needs an interval" unless interval
|
|
176
|
+
return self if @thread
|
|
173
177
|
|
|
174
|
-
@
|
|
178
|
+
wakeup = @wakeup = Thread::Queue.new
|
|
179
|
+
@thread = Thread.new { sweep!(reason: :periodic) until wakeup.pop(timeout: interval) == :stop }
|
|
175
180
|
self
|
|
176
181
|
end
|
|
177
182
|
|
|
178
|
-
# Stops the periodic sweep thread and waits
|
|
179
|
-
|
|
180
|
-
|
|
183
|
+
# Stops the periodic sweep thread and waits up to +timeout+ seconds (nil: for ever) for a sweep
|
|
184
|
+
# in progress to finish - one inspecting sessions of a Chrome that no longer answers can take
|
|
185
|
+
# far longer. A sweep still running then is left to finish on its own; the thread ends after it.
|
|
186
|
+
#
|
|
187
|
+
# @return [Boolean] true when the thread has ended.
|
|
188
|
+
def stop(timeout: DEFAULT_STOP_TIMEOUT)
|
|
189
|
+
return true unless @thread
|
|
181
190
|
|
|
182
191
|
@wakeup << :stop
|
|
183
|
-
|
|
192
|
+
stopped = !@thread.join(timeout).nil?
|
|
193
|
+
Bidi2pdf.logger.warn "chrome_sweeper: a sweep of #{@session_url} still running after #{timeout}s - not waiting for it" unless stopped
|
|
184
194
|
@thread = nil
|
|
195
|
+
stopped
|
|
185
196
|
end
|
|
186
197
|
|
|
187
198
|
private
|
|
@@ -72,7 +72,15 @@ module Bidi2pdf
|
|
|
72
72
|
# pressure and try once more. Default true; false leaves a refused session to the caller.
|
|
73
73
|
attr_accessor :retry_refused_sessions
|
|
74
74
|
|
|
75
|
+
# @return [Numeric, nil] Seconds #shutdown waits, all in all, for its background threads and
|
|
76
|
+
# for closing the warm spares. A Chrome that hangs or ran out of memory answers no command,
|
|
77
|
+
# and every close would wait out its own timeout - an exiting process must not hang on that.
|
|
78
|
+
# What is still running then is left behind (a warning, +session_warmer.shutdown_timeout+);
|
|
79
|
+
# its sessions stay in the registry and a later sweep closes them. +nil+ waits for ever.
|
|
80
|
+
attr_accessor :shutdown_timeout
|
|
81
|
+
|
|
75
82
|
DEFAULT_MAX_IDLE_AGE = 300
|
|
83
|
+
DEFAULT_SHUTDOWN_TIMEOUT = 10
|
|
76
84
|
|
|
77
85
|
def initialize
|
|
78
86
|
@size = 1
|
|
@@ -85,6 +93,7 @@ module Bidi2pdf
|
|
|
85
93
|
@registry_dir = nil
|
|
86
94
|
@sweeper = nil
|
|
87
95
|
@retry_refused_sessions = true
|
|
96
|
+
@shutdown_timeout = DEFAULT_SHUTDOWN_TIMEOUT
|
|
88
97
|
end
|
|
89
98
|
|
|
90
99
|
# The orphan age in seconds, or nil when the sweep is off (also when +:auto+ has no
|
|
@@ -93,12 +102,20 @@ module Bidi2pdf
|
|
|
93
102
|
orphan_age == :auto ? max_idle_age && (max_idle_age * 2) : orphan_age
|
|
94
103
|
end
|
|
95
104
|
|
|
105
|
+
# nil (no limit) or a positive number of seconds - for shutdown_timeout and #shutdown's argument.
|
|
106
|
+
def self.validate_timeout!(name, value)
|
|
107
|
+
return if value.nil? || (value.is_a?(Numeric) && value.positive?)
|
|
108
|
+
|
|
109
|
+
raise ArgumentError, "#{name} must be nil or a positive number of seconds, got #{value.inspect}"
|
|
110
|
+
end
|
|
111
|
+
|
|
96
112
|
# @raise [ArgumentError] if a setting can't be honored - checked once, at construction.
|
|
97
113
|
def validate!
|
|
98
114
|
validate_size!
|
|
99
115
|
validate_max_idle_age!
|
|
100
116
|
validate_orphan_age!
|
|
101
117
|
validate_sweeper!
|
|
118
|
+
validate_shutdown_timeout!
|
|
102
119
|
end
|
|
103
120
|
|
|
104
121
|
private
|
|
@@ -121,6 +138,10 @@ module Bidi2pdf
|
|
|
121
138
|
raise ArgumentError, "orphan_age must be :auto, nil or a positive number of seconds, got #{orphan_age.inspect}"
|
|
122
139
|
end
|
|
123
140
|
|
|
141
|
+
def validate_shutdown_timeout!
|
|
142
|
+
Configuration.validate_timeout!("shutdown_timeout", shutdown_timeout)
|
|
143
|
+
end
|
|
144
|
+
|
|
124
145
|
def validate_sweeper!
|
|
125
146
|
return if sweeper.nil? || sweeper.is_a?(Hash)
|
|
126
147
|
|
|
@@ -273,20 +294,78 @@ module Bidi2pdf
|
|
|
273
294
|
# finish (each of those, seeing @shutdown, retires its own result instead of stashing it - see
|
|
274
295
|
# #stash_or_retire). A subsequent #with_tab still works - it just falls back to a synchronous
|
|
275
296
|
# slot, since checkout never depends on a warm one being there.
|
|
276
|
-
|
|
277
|
-
|
|
297
|
+
#
|
|
298
|
+
# All of that within +timeout+ seconds together (Configuration#shutdown_timeout; nil waits for
|
|
299
|
+
# ever). The spares start closing first, side by side, so a hung sweep cannot use up their share
|
|
300
|
+
# of the deadline. Whatever is still running at the deadline is left to finish on its own, or to
|
|
301
|
+
# die with the process; a spare still closing loses its lease (it keeps its registry entry), so
|
|
302
|
+
# a sweeper can take its session even while this process lives on.
|
|
303
|
+
#
|
|
304
|
+
# @return [Hash{String => Integer}] what was still running at the deadline ("spare" => 1, ...) -
|
|
305
|
+
# empty when everything finished in time.
|
|
306
|
+
# @raise [ArgumentError] for a +timeout+ that is neither nil nor a positive number.
|
|
307
|
+
def shutdown(timeout: @config.shutdown_timeout)
|
|
308
|
+
Configuration.validate_timeout!("timeout", timeout)
|
|
309
|
+
spares, threads = stop_accepting
|
|
310
|
+
deadline = timeout && (now + timeout)
|
|
311
|
+
closers = retire_all(spares)
|
|
312
|
+
pending = (stop_background(threads, deadline) + abandoned_spares(closers, deadline)).tally
|
|
313
|
+
warn_abandoned(pending, timeout) unless pending.empty?
|
|
314
|
+
pending
|
|
315
|
+
end
|
|
316
|
+
|
|
317
|
+
private
|
|
318
|
+
|
|
319
|
+
def stop_accepting
|
|
320
|
+
@mutex.synchronize do
|
|
278
321
|
@shutdown = true
|
|
279
322
|
[@available.dup.tap { @available.clear }, @replenish_threads.dup.tap { @replenish_threads.clear }]
|
|
280
323
|
end
|
|
324
|
+
end
|
|
281
325
|
|
|
326
|
+
# The labels of whatever is still running at the deadline: reaper, sweeper, replenishments.
|
|
327
|
+
def stop_background(threads, deadline)
|
|
282
328
|
@reaper_wakeup << :stop
|
|
283
|
-
@reaper
|
|
284
|
-
@sweeper
|
|
285
|
-
threads
|
|
286
|
-
spares.each { |slot| retire(slot) }
|
|
329
|
+
pending = unfinished([@reaper].compact, deadline, "reaper")
|
|
330
|
+
pending << "sweeper" unless @sweeper.nil? || @sweeper.stop(timeout: remaining(deadline))
|
|
331
|
+
pending + unfinished(threads, deadline, "replenishment")
|
|
287
332
|
end
|
|
288
333
|
|
|
289
|
-
|
|
334
|
+
def remaining(deadline)
|
|
335
|
+
deadline && [deadline - now, 0].max
|
|
336
|
+
end
|
|
337
|
+
|
|
338
|
+
# Thread#join returns nil when the time ran out first.
|
|
339
|
+
def unfinished(threads, deadline, label)
|
|
340
|
+
threads.reject { |thread| thread.join(remaining(deadline)) }.map { label }
|
|
341
|
+
end
|
|
342
|
+
|
|
343
|
+
# [slot, closing thread] pairs.
|
|
344
|
+
def retire_all(slots)
|
|
345
|
+
slots.map do |slot|
|
|
346
|
+
closer = Thread.new do
|
|
347
|
+
retire(slot)
|
|
348
|
+
rescue StandardError => e
|
|
349
|
+
Bidi2pdf.logger.warn "session_warmer: retiring a spare failed: #{e.message}"
|
|
350
|
+
end
|
|
351
|
+
[slot, closer]
|
|
352
|
+
end
|
|
353
|
+
end
|
|
354
|
+
|
|
355
|
+
# Spares still closing at the deadline, with their lease released: the heartbeat would otherwise
|
|
356
|
+
# keep renewing it, and no sweeper takes a leased session.
|
|
357
|
+
def abandoned_spares(closers, deadline)
|
|
358
|
+
closers.reject { |_, closer| closer.join(remaining(deadline)) }.map do |slot, _|
|
|
359
|
+
@registry&.release(session_id_of(slot))
|
|
360
|
+
"spare"
|
|
361
|
+
end
|
|
362
|
+
end
|
|
363
|
+
|
|
364
|
+
# Only called with something pending, which needs a deadline - so +timeout+ is a number here.
|
|
365
|
+
def warn_abandoned(pending, timeout)
|
|
366
|
+
Bidi2pdf.logger.warn "session_warmer: shutdown gave up after #{timeout}s, still running: #{pending.map { |what, count| "#{count} #{what}" }.join(", ")}"
|
|
367
|
+
Bidi2pdf.notification_service.instrument("session_warmer.shutdown_timeout.bidi2pdf", { timeout: timeout, pending: pending })
|
|
368
|
+
end
|
|
290
369
|
|
|
291
370
|
# Fail-fast on purpose (a Chrome that can't start at boot should be loud), but not leaky: if slot
|
|
292
371
|
# N fails, no instance is returned to own slots 1..N-1, so they are retired here first.
|
|
@@ -5,7 +5,18 @@ module Bidi2pdf
|
|
|
5
5
|
module Testcontainers
|
|
6
6
|
class ChromedriverContainer < ::Testcontainers::DockerContainer
|
|
7
7
|
DEFAULT_CHROMEDRIVER_PORT = 3000
|
|
8
|
-
|
|
8
|
+
# Always with a tag: Docker's API pulls every tag of an image named without one, and since
|
|
9
|
+
# every image build is tagged by its commit that is dozens of ~500 MB images - a test run
|
|
10
|
+
# then sits silently in the pull for many minutes.
|
|
11
|
+
DEFAULT_IMAGE = "dieters877565/chromedriver:latest"
|
|
12
|
+
|
|
13
|
+
# BIDI2PDF_BUILD_CHROMEDRIVER_IMAGE=true: build the image from the checkout (build_dir,
|
|
14
|
+
# docker_file) and run that, so a change to the image is tested before it is published -
|
|
15
|
+
# bidi2pdf's own CI sets it. Unset, the published image is pulled. Testcontainers' #start
|
|
16
|
+
# always pulls, so a locally built tag cannot go through it; #start_local_image builds instead.
|
|
17
|
+
def self.build_locally?(env = ENV)
|
|
18
|
+
env["BIDI2PDF_BUILD_CHROMEDRIVER_IMAGE"] == "true"
|
|
19
|
+
end
|
|
9
20
|
|
|
10
21
|
attr_reader :docker_file, :build_dir
|
|
11
22
|
|
|
@@ -132,7 +132,7 @@ def start_chromedriver_container(build_dir:, mounts:, shared_network:, chromedri
|
|
|
132
132
|
|
|
133
133
|
container.with_filesystem_binds(mounts) if mounts&.any?
|
|
134
134
|
|
|
135
|
-
container.start
|
|
135
|
+
ChromedriverTestcontainer.build_locally? ? container.start_local_image : container.start
|
|
136
136
|
|
|
137
137
|
container
|
|
138
138
|
end
|
data/lib/bidi2pdf/version.rb
CHANGED
|
@@ -133,11 +133,18 @@ module Bidi2pdf
|
|
|
133
133
|
# @return [Result]
|
|
134
134
|
def sweep!: (?reason: ::Symbol, ?check_interval: untyped?, ?pressure: bool) -> untyped
|
|
135
135
|
|
|
136
|
-
# Starts the periodic sweep thread (needs an +interval+). Idempotent.
|
|
136
|
+
# Starts the periodic sweep thread (needs an +interval+). Idempotent. Each thread has its own
|
|
137
|
+
# wakeup queue, so one that #stop stopped waiting for can never take a later thread's :stop.
|
|
138
|
+
# After such a #stop, a new thread can sweep while the abandoned sweep is still running: the
|
|
139
|
+
# lock file lets only one of them sweep at a time, the other's sweep is skipped.
|
|
137
140
|
def start: () -> self
|
|
138
141
|
|
|
139
|
-
# Stops the periodic sweep thread and waits
|
|
140
|
-
|
|
142
|
+
# Stops the periodic sweep thread and waits up to +timeout+ seconds (nil: for ever) for a sweep
|
|
143
|
+
# in progress to finish - one inspecting sessions of a Chrome that no longer answers can take
|
|
144
|
+
# far longer. A sweep still running then is left to finish on its own; the thread ends after it.
|
|
145
|
+
#
|
|
146
|
+
# @return [Boolean] true when the thread has ended.
|
|
147
|
+
def stop: (?timeout: untyped) -> (true | untyped)
|
|
141
148
|
|
|
142
149
|
private
|
|
143
150
|
|
|
@@ -72,6 +72,8 @@ module Bidi2pdf
|
|
|
72
72
|
|
|
73
73
|
@retry_refused_sessions: untyped
|
|
74
74
|
|
|
75
|
+
@shutdown_timeout: untyped
|
|
76
|
+
|
|
75
77
|
# @return [Integer] Number of Chrome slots to keep pre-warmed.
|
|
76
78
|
attr_accessor size: untyped
|
|
77
79
|
|
|
@@ -120,14 +122,26 @@ module Bidi2pdf
|
|
|
120
122
|
# pressure and try once more. Default true; false leaves a refused session to the caller.
|
|
121
123
|
attr_accessor retry_refused_sessions: untyped
|
|
122
124
|
|
|
125
|
+
# @return [Numeric, nil] Seconds #shutdown waits, all in all, for its background threads and
|
|
126
|
+
# for closing the warm spares. A Chrome that hangs or ran out of memory answers no command,
|
|
127
|
+
# and every close would wait out its own timeout - an exiting process must not hang on that.
|
|
128
|
+
# What is still running then is left behind (a warning, +session_warmer.shutdown_timeout+);
|
|
129
|
+
# its sessions stay in the registry and a later sweep closes them. +nil+ waits for ever.
|
|
130
|
+
attr_accessor shutdown_timeout: untyped
|
|
131
|
+
|
|
123
132
|
DEFAULT_MAX_IDLE_AGE: 300
|
|
124
133
|
|
|
134
|
+
DEFAULT_SHUTDOWN_TIMEOUT: 10
|
|
135
|
+
|
|
125
136
|
def initialize: () -> void
|
|
126
137
|
|
|
127
138
|
# The orphan age in seconds, or nil when the sweep is off (also when +:auto+ has no
|
|
128
139
|
# max_idle_age to derive it from).
|
|
129
140
|
def effective_orphan_age: () -> untyped
|
|
130
141
|
|
|
142
|
+
# nil (no limit) or a positive number of seconds - for shutdown_timeout and #shutdown's argument.
|
|
143
|
+
def self.validate_timeout!: (untyped name, untyped value) -> (nil | untyped)
|
|
144
|
+
|
|
131
145
|
# @raise [ArgumentError] if a setting can't be honored - checked once, at construction.
|
|
132
146
|
def validate!: () -> untyped
|
|
133
147
|
|
|
@@ -139,6 +153,8 @@ module Bidi2pdf
|
|
|
139
153
|
|
|
140
154
|
def validate_orphan_age!: () -> (nil | untyped)
|
|
141
155
|
|
|
156
|
+
def validate_shutdown_timeout!: () -> untyped
|
|
157
|
+
|
|
142
158
|
def validate_sweeper!: () -> (nil | untyped)
|
|
143
159
|
end
|
|
144
160
|
|
|
@@ -204,10 +220,40 @@ module Bidi2pdf
|
|
|
204
220
|
# finish (each of those, seeing @shutdown, retires its own result instead of stashing it - see
|
|
205
221
|
# #stash_or_retire). A subsequent #with_tab still works - it just falls back to a synchronous
|
|
206
222
|
# slot, since checkout never depends on a warm one being there.
|
|
207
|
-
|
|
223
|
+
#
|
|
224
|
+
# All of that within +timeout+ seconds together (Configuration#shutdown_timeout; nil waits for
|
|
225
|
+
# ever). The spares start closing first, side by side, so a hung sweep cannot use up their share
|
|
226
|
+
# of the deadline. Whatever is still running at the deadline is left to finish on its own, or to
|
|
227
|
+
# die with the process; a spare still closing loses its lease (it keeps its registry entry), so
|
|
228
|
+
# a sweeper can take its session even while this process lives on.
|
|
229
|
+
#
|
|
230
|
+
# @return [Hash{String => Integer}] what was still running at the deadline ("spare" => 1, ...) -
|
|
231
|
+
# empty when everything finished in time.
|
|
232
|
+
# @raise [ArgumentError] for a +timeout+ that is neither nil nor a positive number.
|
|
233
|
+
def shutdown: (?timeout: untyped) -> untyped
|
|
208
234
|
|
|
209
235
|
private
|
|
210
236
|
|
|
237
|
+
def stop_accepting: () -> untyped
|
|
238
|
+
|
|
239
|
+
# The labels of whatever is still running at the deadline: reaper, sweeper, replenishments.
|
|
240
|
+
def stop_background: (untyped threads, untyped deadline) -> untyped
|
|
241
|
+
|
|
242
|
+
def remaining: (untyped deadline) -> untyped
|
|
243
|
+
|
|
244
|
+
# Thread#join returns nil when the time ran out first.
|
|
245
|
+
def unfinished: (untyped threads, untyped deadline, untyped label) -> untyped
|
|
246
|
+
|
|
247
|
+
# [slot, closing thread] pairs.
|
|
248
|
+
def retire_all: (untyped slots) -> untyped
|
|
249
|
+
|
|
250
|
+
# Spares still closing at the deadline, with their lease released: the heartbeat would otherwise
|
|
251
|
+
# keep renewing it, and no sweeper takes a leased session.
|
|
252
|
+
def abandoned_spares: (untyped closers, untyped deadline) -> untyped
|
|
253
|
+
|
|
254
|
+
# Only called with something pending, which needs a deadline - so +timeout+ is a number here.
|
|
255
|
+
def warn_abandoned: (untyped pending, untyped timeout) -> untyped
|
|
256
|
+
|
|
211
257
|
# Fail-fast on purpose (a Chrome that can't start at boot should be loud), but not leaky: if slot
|
|
212
258
|
# N fails, no instance is returned to own slots 1..N-1, so they are retired here first.
|
|
213
259
|
def prewarm: () -> untyped
|
|
@@ -18,7 +18,16 @@ module Bidi2pdf
|
|
|
18
18
|
|
|
19
19
|
DEFAULT_CHROMEDRIVER_PORT: 3000
|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
# Always with a tag: Docker's API pulls every tag of an image named without one, and since
|
|
22
|
+
# every image build is tagged by its commit that is dozens of ~500 MB images - a test run
|
|
23
|
+
# then sits silently in the pull for many minutes.
|
|
24
|
+
DEFAULT_IMAGE: "dieters877565/chromedriver:latest"
|
|
25
|
+
|
|
26
|
+
# BIDI2PDF_BUILD_CHROMEDRIVER_IMAGE=true: build the image from the checkout (build_dir,
|
|
27
|
+
# docker_file) and run that, so a change to the image is tested before it is published -
|
|
28
|
+
# bidi2pdf's own CI sets it. Unset, the published image is pulled. Testcontainers' #start
|
|
29
|
+
# always pulls, so a locally built tag cannot go through it; #start_local_image builds instead.
|
|
30
|
+
def self.build_locally?: (?untyped env) -> untyped
|
|
22
31
|
|
|
23
32
|
attr_reader docker_file: untyped
|
|
24
33
|
|
data/sig/bidi2pdf/version.rbs
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: bidi2pdf
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Dieter S.
|
|
@@ -408,8 +408,10 @@ files:
|
|
|
408
408
|
- docker/Dockerfile.chromedriver
|
|
409
409
|
- docker/Dockerfile.slim
|
|
410
410
|
- docker/chrome-driver/chrome.json
|
|
411
|
+
- docker/chromedriver-watchdog.sh
|
|
411
412
|
- docker/docker-compose.yml
|
|
412
413
|
- docker/entrypoint.sh
|
|
414
|
+
- docker/install-hyphen-data.sh
|
|
413
415
|
- docker/nginx/default.conf
|
|
414
416
|
- docker/nginx/htpasswd
|
|
415
417
|
- exe/bidi2pdf
|