@link-assistant/web-capture 1.4.2 → 1.7.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (18) hide show
  1. package/CHANGELOG.md +47 -0
  2. package/Dockerfile +26 -21
  3. package/README.md +92 -72
  4. package/bin/web-capture.js +259 -51
  5. package/package.json +5 -1
  6. package/src/archive.js +19 -6
  7. package/src/batch.js +3 -3
  8. package/src/extract-images.js +120 -0
  9. package/src/gdocs.js +8 -6
  10. package/src/lib.js +57 -2
  11. package/src/markdown.js +7 -2
  12. package/src/postprocess.js +2 -4
  13. package/tests/unit/batch.test.js +4 -4
  14. package/tests/unit/extract-images.test.js +233 -0
  15. package/tests/unit/gdocs.test.js +75 -0
  16. package/tests/unit/heading-image.test.js +49 -0
  17. package/tests/unit/postprocess.test.js +4 -2
  18. package/tests/unit/publish-verification.test.js +57 -0
package/CHANGELOG.md CHANGED
@@ -1,5 +1,52 @@
1
1
  # @link-assistant/web-capture
2
2
 
3
+ ## 1.7.2
4
+
5
+ ### Patch Changes
6
+
7
+ - 9c0df3c: Retry npm publish verification long enough to tolerate registry propagation delays after a successful publish.
8
+
9
+ ## 1.7.1
10
+
11
+ ### Patch Changes
12
+
13
+ - 3ee464b: Add heading-image parity guard rail tests to prevent <img> inside headings from being silently dropped during HTML-to-Markdown conversion
14
+
15
+ ## 1.7.0
16
+
17
+ ### Minor Changes
18
+
19
+ - b6d77d6: Fix archive quality bugs: relative image paths in markdown, HTML pretty-printing, entity decoding, and rename article to document filenames
20
+
21
+ ## 1.5.1
22
+
23
+ ### Patch Changes
24
+
25
+ - e6dea94: Fix Google Docs capture: archive produces real zip, default markdown extracts images to files instead of stripping
26
+
27
+ ## 1.5.0
28
+
29
+ ### Minor Changes
30
+
31
+ - a463061: Markdown mode now keeps original remote image URLs and strips base64 by default (single-file output).
32
+ Archive mode downloads images to images/ folder by default (keepOriginalLinks=false, embedImages=false).
33
+ API and CLI share identical defaults for all modes.
34
+ Add --embed-images, --images-dir, --no-extract-images, --archive, --data-dir, --keep-original-links flags.
35
+ Add embedImages and keepOriginalLinks query params to /markdown and /archive API endpoints.
36
+ Change /markdown API defaults: keepOriginalLinks=true, embedImages=false.
37
+ Remove --enhanced umbrella flag; make sub-features default to true.
38
+ Change default format from html to markdown.
39
+ Auto-derive output directory from URL when -o is omitted.
40
+ Use content-hash filenames for extracted images instead of positional numbering.
41
+ Replace process.env ternaries with lino-arguments getenv() helper.
42
+ Add `WEB_CAPTURE_*` environment variable support for all feature flags.
43
+
44
+ ## 1.4.3
45
+
46
+ ### Patch Changes
47
+
48
+ - c9d1132: Fix cargo publish token resolution: fallback to CARGO_TOKEN when CARGO_REGISTRY_TOKEN is not set, and fail instead of silently skipping publish
49
+
3
50
  ## 1.4.2
4
51
 
5
52
  ### Patch Changes
package/Dockerfile CHANGED
@@ -3,27 +3,32 @@ FROM node:22-bullseye
3
3
  WORKDIR /app
4
4
 
5
5
  # Install dependencies for headless Chrome (used by both Puppeteer and Playwright)
6
- RUN apt-get update && apt-get install -y \
7
- chromium \
8
- fonts-liberation \
9
- libasound2 \
10
- libatk-bridge2.0-0 \
11
- libatk1.0-0 \
12
- libatspi2.0-0 \
13
- libcups2 \
14
- libdbus-1-3 \
15
- libdrm2 \
16
- libgbm1 \
17
- libgtk-3-0 \
18
- libnspr4 \
19
- libnss3 \
20
- libxcomposite1 \
21
- libxdamage1 \
22
- libxfixes3 \
23
- libxrandr2 \
24
- libxshmfence1 \
25
- xdg-utils \
26
- --no-install-recommends && \
6
+ # Retry loop handles transient network errors from Debian mirrors in CI
7
+ RUN apt-get update && \
8
+ for i in 1 2 3; do \
9
+ apt-get install -y \
10
+ chromium \
11
+ fonts-liberation \
12
+ libasound2 \
13
+ libatk-bridge2.0-0 \
14
+ libatk1.0-0 \
15
+ libatspi2.0-0 \
16
+ libcups2 \
17
+ libdbus-1-3 \
18
+ libdrm2 \
19
+ libgbm1 \
20
+ libgtk-3-0 \
21
+ libnspr4 \
22
+ libnss3 \
23
+ libxcomposite1 \
24
+ libxdamage1 \
25
+ libxfixes3 \
26
+ libxrandr2 \
27
+ libxshmfence1 \
28
+ xdg-utils \
29
+ --no-install-recommends && break || \
30
+ { echo "apt-get install attempt $i failed, retrying..."; sleep 5; }; \
31
+ done && \
27
32
  rm -rf /var/lib/apt/lists/*
28
33
 
29
34
  # Set Puppeteer environment variables
package/README.md CHANGED
@@ -8,8 +8,8 @@
8
8
 
9
9
  A CLI and microservice to fetch URLs and render them as:
10
10
 
11
+ - **Markdown**: Converted from HTML with image extraction (default)
11
12
  - **HTML**: Rendered page content
12
- - **Markdown**: Converted from HTML
13
13
  - **PNG/JPEG screenshot**: Viewport or full-page capture with theme support
14
14
  - **ZIP archive**: Markdown + locally downloaded images
15
15
  - **PDF**: Print-quality document with embedded images
@@ -28,8 +28,6 @@ npm install -g @link-assistant/web-capture
28
28
  ```bash
29
29
  cd js
30
30
  npm install
31
- # or
32
- yarn install
33
31
  ```
34
32
 
35
33
  ## Quick Start
@@ -37,32 +35,52 @@ yarn install
37
35
  ### CLI Usage
38
36
 
39
37
  ```bash
40
- # Capture a URL as HTML (output to stdout)
38
+ # Capture a URL as Markdown (default format)
39
+ # Output auto-derived to ./data/web-capture/<host>/<path>/document.md
41
40
  web-capture https://example.com
42
41
 
43
- # Capture as Markdown and save to file
44
- web-capture https://example.com --format markdown --output page.md
42
+ # Capture as Markdown to a specific file
43
+ web-capture https://example.com -o page.md
44
+
45
+ # Write to stdout explicitly
46
+ web-capture https://example.com -o -
47
+
48
+ # Capture as HTML
49
+ web-capture https://example.com --format html -o page.html
45
50
 
46
51
  # Take a PNG screenshot
47
- web-capture https://example.com --format png --output screenshot.png
52
+ web-capture https://example.com --format png -o screenshot.png
48
53
 
49
54
  # Take a JPEG screenshot with custom quality
50
- web-capture https://example.com --format jpeg --quality 60 --output screenshot.jpg
55
+ web-capture https://example.com --format jpeg --quality 60 -o screenshot.jpg
51
56
 
52
57
  # Dark theme full-page screenshot
53
- web-capture https://example.com --format png --theme dark --fullPage --output dark.png
58
+ web-capture https://example.com --format png --theme dark --fullPage -o dark.png
54
59
 
55
60
  # Custom viewport size
56
61
  web-capture https://example.com --format png --width 1920 --height 1080 -o wide.png
57
62
 
58
63
  # Download as PDF
59
- web-capture https://example.com --format pdf --output page.pdf
64
+ web-capture https://example.com --format pdf -o page.pdf
60
65
 
61
66
  # Download as DOCX
62
- web-capture https://example.com --format docx --output page.docx
67
+ web-capture https://example.com --format docx -o page.docx
68
+
69
+ # Create a ZIP archive (default: zip format)
70
+ web-capture https://example.com --archive
71
+
72
+ # Create a ZIP archive with specific format
73
+ web-capture https://example.com --archive zip -o site.zip
74
+ web-capture https://example.com --archive tar.gz -o site.tar.gz
75
+
76
+ # Keep images inline as base64 (opt-in)
77
+ web-capture https://example.com --embed-images -o page.md
63
78
 
64
- # Download as ZIP archive with local images
65
- web-capture https://example.com --format archive --output site.zip
79
+ # Custom images directory
80
+ web-capture https://example.com --images-dir assets -o page.md
81
+
82
+ # Disable specific features
83
+ web-capture https://example.com --no-extract-latex --no-post-process -o page.md
66
84
 
67
85
  # Start as API server
68
86
  web-capture --serve
@@ -71,34 +89,14 @@ web-capture --serve
71
89
  web-capture --serve --port 8080
72
90
  ```
73
91
 
74
- ### Downloading Habr Articles
75
-
76
- The service is tested against real Habr.com articles. Here are examples for each format:
92
+ ### Google Docs
77
93
 
78
94
  ```bash
79
- # Markdown (default: remote image links)
80
- web-capture https://habr.com/en/articles/895896/ -f markdown -o article.md
81
-
82
- # ZIP archive with local images (markdown format, default)
83
- web-capture https://habr.com/en/articles/895896/ -f archive -o article.zip
84
-
85
- # ZIP archive with local images (HTML format with CSS)
86
- web-capture https://habr.com/en/articles/895896/ -f archive --documentFormat html -o article-html.zip
95
+ # Capture public Google Doc as Markdown
96
+ web-capture https://docs.google.com/document/d/DOC_ID/edit
87
97
 
88
- # Light theme full-page PNG
89
- web-capture https://habr.com/en/articles/895896/ -f png --theme light --fullPage -o light.png
90
-
91
- # Dark theme full-page PNG
92
- web-capture https://habr.com/en/articles/895896/ -f png --theme dark --fullPage -o dark.png
93
-
94
- # JPEG with 90% quality
95
- web-capture https://habr.com/en/articles/895896/ -f jpeg --quality 90 -o article.jpg
96
-
97
- # PDF with dark theme
98
- web-capture https://habr.com/en/articles/895896/ -f pdf --theme dark -o article.pdf
99
-
100
- # DOCX with embedded images
101
- web-capture https://habr.com/en/articles/895896/ -f docx -o article.docx
98
+ # Capture private Google Doc with API token
99
+ web-capture https://docs.google.com/document/d/DOC_ID/edit --apiToken YOUR_TOKEN
102
100
  ```
103
101
 
104
102
  ## API Endpoints (Server Mode)
@@ -124,11 +122,13 @@ Returns the raw HTML content of the specified URL.
124
122
  GET /markdown?url=<URL>
125
123
  ```
126
124
 
127
- Converts the HTML content of the specified URL to Markdown format.
125
+ Converts the HTML content of the specified URL to Markdown format. By default, original remote image URLs are preserved and base64 data URIs are stripped (clean single-file output). Use `keepOriginalLinks=false` to strip all images, or `embedImages=true` to keep base64 images inline.
128
126
 
129
- | Parameter | Required | Description | Default |
130
- | --------- | -------- | ------------ | ------- |
131
- | `url` | Yes | URL to fetch | - |
127
+ | Parameter | Required | Description | Default |
128
+ | ------------------- | -------- | ------------------------------------------ | ------- |
129
+ | `url` | Yes | URL to fetch | - |
130
+ | `embedImages` | No | Keep base64 images inline (`true`/`false`) | `false` |
131
+ | `keepOriginalLinks` | No | Keep original remote URLs, strip base64 | `true` |
132
132
 
133
133
  ### Image Endpoint
134
134
 
@@ -156,19 +156,21 @@ Returns a screenshot of the specified URL.
156
156
  GET /archive?url=<URL>&localImages=true&documentFormat=markdown
157
157
  ```
158
158
 
159
- Returns a ZIP archive containing either `article.md` or `article.html` and asset directories (`images/`, `css/`).
159
+ Returns a ZIP archive containing either `document.md` or `document.html` and asset directories (`images/`, `css/`).
160
160
 
161
- | Parameter | Required | Description | Default |
162
- | ---------------- | -------- | ---------------------------------------- | ---------- |
163
- | `url` | Yes | URL to archive | - |
164
- | `localImages` | No | Download images locally into the archive | true |
165
- | `documentFormat` | No | Document format: `markdown` or `html` | `markdown` |
161
+ | Parameter | Required | Description | Default |
162
+ | ------------------- | -------- | -------------------------------------------- | ---------- |
163
+ | `url` | Yes | URL to archive | - |
164
+ | `localImages` | No | Download images locally into the archive | `true` |
165
+ | `documentFormat` | No | Document format: `markdown` or `html` | `markdown` |
166
+ | `embedImages` | No | Keep base64 images inline in the document | `false` |
167
+ | `keepOriginalLinks` | No | Keep original remote URLs (skip downloading) | `false` |
166
168
 
167
169
  **Archive structure** (with `localImages=true`):
168
170
 
169
171
  ```
170
172
  archive.zip
171
- ├── article.md # or article.html when documentFormat=html
173
+ ├── document.md # or document.html when documentFormat=html
172
174
  ├── images/
173
175
  │ ├── image-1.png
174
176
  │ ├── image-2.jpg
@@ -235,25 +237,39 @@ web-capture --serve [--port <port>]
235
237
  web-capture <url> [options]
236
238
  ```
237
239
 
238
- | Option | Short | Description | Default |
239
- | ------------------ | ----- | ---------------------------------------------- | ---------------------------------------- |
240
- | `--format` | `-f` | Output format (see below) | `html` |
241
- | `--output` | `-o` | Output file path | stdout (text) or auto-generated (images) |
242
- | `--engine` | `-e` | Browser engine: `puppeteer`, `playwright` | `puppeteer` (or BROWSER_ENGINE env) |
243
- | `--theme` | `-t` | Color scheme: `light`, `dark`, `no-preference` | browser default |
244
- | `--width` | | Viewport width in pixels | 1280 |
245
- | `--height` | | Viewport height in pixels | 800 |
246
- | `--quality` | | JPEG quality 0-100 | 80 |
247
- | `--fullPage` | | Capture full scrollable page | false |
248
- | `--localImages` | | Download images locally in archive mode | true |
249
- | `--documentFormat` | | Document format in archive: `markdown`, `html` | `markdown` |
240
+ | Option | Short | Description | Default |
241
+ | --------------------------- | ----- | ---------------------------------------------- | ----------------------------------- |
242
+ | `--format` | `-f` | Output format (see below) | `markdown` |
243
+ | `--output` | `-o` | Output file path. Use `-o -` for stdout | auto-derived from URL |
244
+ | `--data-dir` | | Base directory for auto-derived output paths | `./data/web-capture` |
245
+ | `--engine` | `-e` | Browser engine: `puppeteer`, `playwright` | `puppeteer` (or BROWSER_ENGINE env) |
246
+ | `--theme` | `-t` | Color scheme: `light`, `dark`, `no-preference` | browser default |
247
+ | `--width` | | Viewport width in pixels | 1280 |
248
+ | `--height` | | Viewport height in pixels | 800 |
249
+ | `--quality` | | JPEG quality 0-100 | 80 |
250
+ | `--fullPage` | | Capture full scrollable page | false |
251
+ | `--embed-images` | | Keep images as inline base64 data URIs | false |
252
+ | `--no-extract-images` | | Alias for `--embed-images` | false |
253
+ | `--keep-original-links` | | Keep original remote URLs, strip base64 | false |
254
+ | `--images-dir` | | Subdirectory name for extracted images | `images` |
255
+ | `--archive` | | Create archive: `zip`, `7z`, `tar.gz`, `tar` | - |
256
+ | `--document-format` | | Document format in archive: `markdown`, `html` | `markdown` |
257
+ | `--localImages` | | Download images locally in archive mode | true |
258
+ | `--extract-latex` | | Extract LaTeX formulas | true |
259
+ | `--no-extract-latex` | | Disable LaTeX extraction | - |
260
+ | `--extract-metadata` | | Extract article metadata | true |
261
+ | `--no-extract-metadata` | | Disable metadata extraction | - |
262
+ | `--post-process` | | Apply post-processing | true |
263
+ | `--no-post-process` | | Disable post-processing | - |
264
+ | `--detect-code-language` | | Detect code block languages | true |
265
+ | `--no-detect-code-language` | | Disable code language detection | - |
250
266
 
251
267
  **Supported formats:**
252
268
 
253
269
  | Format | Description |
254
270
  | ----------------- | ---------------------------------------- |
255
- | `html` | Rendered HTML (default) |
256
- | `markdown` / `md` | Markdown conversion |
271
+ | `markdown` / `md` | Markdown conversion (default) |
272
+ | `html` | Rendered HTML |
257
273
  | `image` / `png` | PNG screenshot (lossless) |
258
274
  | `jpeg` | JPEG screenshot (configurable quality) |
259
275
  | `pdf` | PDF with embedded images |
@@ -271,15 +287,19 @@ Configuration values are resolved with the following priority (highest to lowest
271
287
 
272
288
  ### Environment Variables
273
289
 
274
- ```bash
275
- # Set port via environment variable
276
- export PORT=8080
277
- web-capture --serve
278
-
279
- # Set browser engine
280
- export BROWSER_ENGINE=playwright
281
- web-capture https://example.com --format png
282
- ```
290
+ | Variable | Description | Default |
291
+ | ---------------------------------- | ----------------------------------- | -------------------- |
292
+ | `PORT` | Server port | `3000` |
293
+ | `BROWSER_ENGINE` | Browser engine | `puppeteer` |
294
+ | `API_TOKEN` | API token for authenticated capture | - |
295
+ | `WEB_CAPTURE_DATA_DIR` | Base directory for output | `./data/web-capture` |
296
+ | `WEB_CAPTURE_EMBED_IMAGES` | `0`/`1` — keep images inline | `0` |
297
+ | `WEB_CAPTURE_KEEP_ORIGINAL_LINKS` | `0`/`1` — keep original remote URLs | `0` |
298
+ | `WEB_CAPTURE_IMAGES_DIR` | Subdirectory for extracted images | `images` |
299
+ | `WEB_CAPTURE_EXTRACT_LATEX` | `0`/`1` — extract LaTeX | `1` |
300
+ | `WEB_CAPTURE_EXTRACT_METADATA` | `0`/`1` — extract metadata | `1` |
301
+ | `WEB_CAPTURE_POST_PROCESS` | `0`/`1` — post-processing | `1` |
302
+ | `WEB_CAPTURE_DETECT_CODE_LANGUAGE` | `0`/`1` — detect code langs | `1` |
283
303
 
284
304
  ## Browser Engine Support
285
305