@link-assistant/web-capture 1.1.2 → 1.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +57 -13
- package/Dockerfile +26 -21
- package/README.md +290 -160
- package/bin/web-capture.js +622 -17
- package/package.json +15 -4
- package/src/animation.js +283 -0
- package/src/archive.js +225 -0
- package/src/batch.js +180 -0
- package/src/browser.js +150 -111
- package/src/docx.js +182 -0
- package/src/extract-images.js +120 -0
- package/src/figures.js +208 -0
- package/src/gdocs.js +335 -0
- package/src/image.js +65 -8
- package/src/index.js +15 -0
- package/src/latex.js +127 -0
- package/src/lib.js +281 -1
- package/src/localize-images.js +202 -0
- package/src/markdown.js +7 -2
- package/src/metadata.js +333 -0
- package/src/pdf.js +76 -0
- package/src/popups.js +171 -0
- package/src/postprocess.js +263 -0
- package/src/retry.js +48 -0
- package/src/themed-image.js +167 -0
- package/src/verify.js +312 -0
- package/tests/e2e/docker.test.js +51 -12
- package/tests/e2e/process.test.js +34 -10
- package/tests/fixtures/habr-articles.js +45 -0
- package/tests/integration/api-endpoints.test.js +231 -0
- package/tests/integration/habr-article.test.js +337 -0
- package/tests/mock/index.test.js +5 -3
- package/tests/unit/animation.test.js +33 -0
- package/tests/unit/batch.test.js +114 -0
- package/tests/unit/browser.test.js +38 -2
- package/tests/unit/extract-images.test.js +233 -0
- package/tests/unit/figures.test.js +74 -0
- package/tests/unit/gdocs.test.js +278 -0
- package/tests/unit/heading-image.test.js +49 -0
- package/tests/unit/latex.test.js +123 -0
- package/tests/unit/localize-images.test.js +78 -0
- package/tests/unit/metadata.test.js +140 -0
- package/tests/unit/postprocess.test.js +110 -0
- package/tests/unit/retry.test.js +111 -0
- package/tests/unit/verify.test.js +182 -0
- package/.github/workflows/release.yml +0 -294
- package/ARCHITECTURE.md +0 -982
- package/LICENSE +0 -24
- package/apply.sh +0 -166
- package/docs/market-analysis-2025-05-13.md +0 -64
- package/scripts/changeset-version.mjs +0 -38
- package/scripts/create-github-release.mjs +0 -93
- package/scripts/create-manual-changeset.mjs +0 -88
- package/scripts/format-github-release.mjs +0 -83
- package/scripts/format-release-notes.mjs +0 -218
- package/scripts/instant-version-bump.mjs +0 -121
- package/scripts/publish-to-npm.mjs +0 -128
- package/scripts/setup-npm.mjs +0 -37
- package/scripts/validate-changeset.mjs +0 -106
- package/scripts/version-and-commit.mjs +0 -237
package/CHANGELOG.md
CHANGED
|
@@ -1,22 +1,66 @@
|
|
|
1
|
-
# web-capture
|
|
1
|
+
# @link-assistant/web-capture
|
|
2
2
|
|
|
3
|
-
## 1.1
|
|
3
|
+
## 1.7.1
|
|
4
4
|
|
|
5
5
|
### Patch Changes
|
|
6
6
|
|
|
7
|
-
-
|
|
7
|
+
- 3ee464b: Add heading-image parity guard rail tests to prevent <img> inside headings from being silently dropped during HTML-to-Markdown conversion
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
## 1.7.0
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
### Minor Changes
|
|
12
|
+
|
|
13
|
+
- b6d77d6: Fix archive quality bugs: relative image paths in markdown, HTML pretty-printing, entity decoding, and rename article to document filenames
|
|
14
|
+
|
|
15
|
+
## 1.5.1
|
|
16
|
+
|
|
17
|
+
### Patch Changes
|
|
18
|
+
|
|
19
|
+
- e6dea94: Fix Google Docs capture: archive produces real zip, default markdown extracts images to files instead of stripping
|
|
20
|
+
|
|
21
|
+
## 1.5.0
|
|
22
|
+
|
|
23
|
+
### Minor Changes
|
|
24
|
+
|
|
25
|
+
- a463061: Markdown mode now keeps original remote image URLs and strips base64 by default (single-file output).
|
|
26
|
+
Archive mode downloads images to images/ folder by default (keepOriginalLinks=false, embedImages=false).
|
|
27
|
+
API and CLI share identical defaults for all modes.
|
|
28
|
+
Add --embed-images, --images-dir, --no-extract-images, --archive, --data-dir, --keep-original-links flags.
|
|
29
|
+
Add embedImages and keepOriginalLinks query params to /markdown and /archive API endpoints.
|
|
30
|
+
Change /markdown API defaults: keepOriginalLinks=true, embedImages=false.
|
|
31
|
+
Remove --enhanced umbrella flag; make sub-features default to true.
|
|
32
|
+
Change default format from html to markdown.
|
|
33
|
+
Auto-derive output directory from URL when -o is omitted.
|
|
34
|
+
Use content-hash filenames for extracted images instead of positional numbering.
|
|
35
|
+
Replace process.env ternaries with lino-arguments getenv() helper.
|
|
36
|
+
Add `WEB_CAPTURE_*` environment variable support for all feature flags.
|
|
37
|
+
|
|
38
|
+
## 1.4.3
|
|
12
39
|
|
|
13
40
|
### Patch Changes
|
|
14
41
|
|
|
15
|
-
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
42
|
+
- c9d1132: Fix cargo publish token resolution: fallback to CARGO_TOKEN when CARGO_REGISTRY_TOKEN is not set, and fail instead of silently skipping publish
|
|
43
|
+
|
|
44
|
+
## 1.4.2
|
|
45
|
+
|
|
46
|
+
### Patch Changes
|
|
47
|
+
|
|
48
|
+
- 94c1953: Update lino-arguments to ^0.3.0, fix npm publish OIDC fallback, and align CI/CD workflows with reference repo best practices
|
|
49
|
+
|
|
50
|
+
## 1.4.1
|
|
51
|
+
|
|
52
|
+
### Patch Changes
|
|
53
|
+
|
|
54
|
+
- 4570a2b: Fix CI/CD release pipeline: resolve git show path bug in version-and-commit.mjs where `git show origin/main:package.json` failed because git show uses repo-root-relative paths (should be `js/package.json`). Add npx-based fallback in setup-npm.mjs for Node.js 22.22.2 broken npm issue.
|
|
55
|
+
|
|
56
|
+
## 1.4.0
|
|
57
|
+
|
|
58
|
+
### Minor Changes
|
|
59
|
+
|
|
60
|
+
- f464988: Add Google Docs document to Markdown capture support with API token authentication
|
|
61
|
+
|
|
62
|
+
## 1.3.0
|
|
63
|
+
|
|
64
|
+
### Minor Changes
|
|
65
|
+
|
|
66
|
+
- 141458e: Implement all web capture best practices from meta-theory reference scripts (R1-R7): LaTeX formula extraction (Habr, KaTeX, MathJax), article metadata extraction, markdown post-processing pipeline, animation capture with loop detection, dual-theme screenshots, figure image extraction, markdown image localization, content verification with fuzzy matching, and batch processing with JSON/MJS configuration. Add enhanced HTML-to-markdown conversion with configurable options, 3 new API endpoints (/animation, /figures, /themed-image), and expanded CLI flags.
|
package/Dockerfile
CHANGED
|
@@ -3,27 +3,32 @@ FROM node:22-bullseye
|
|
|
3
3
|
WORKDIR /app
|
|
4
4
|
|
|
5
5
|
# Install dependencies for headless Chrome (used by both Puppeteer and Playwright)
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
6
|
+
# Retry loop handles transient network errors from Debian mirrors in CI
|
|
7
|
+
RUN apt-get update && \
|
|
8
|
+
for i in 1 2 3; do \
|
|
9
|
+
apt-get install -y \
|
|
10
|
+
chromium \
|
|
11
|
+
fonts-liberation \
|
|
12
|
+
libasound2 \
|
|
13
|
+
libatk-bridge2.0-0 \
|
|
14
|
+
libatk1.0-0 \
|
|
15
|
+
libatspi2.0-0 \
|
|
16
|
+
libcups2 \
|
|
17
|
+
libdbus-1-3 \
|
|
18
|
+
libdrm2 \
|
|
19
|
+
libgbm1 \
|
|
20
|
+
libgtk-3-0 \
|
|
21
|
+
libnspr4 \
|
|
22
|
+
libnss3 \
|
|
23
|
+
libxcomposite1 \
|
|
24
|
+
libxdamage1 \
|
|
25
|
+
libxfixes3 \
|
|
26
|
+
libxrandr2 \
|
|
27
|
+
libxshmfence1 \
|
|
28
|
+
xdg-utils \
|
|
29
|
+
--no-install-recommends && break || \
|
|
30
|
+
{ echo "apt-get install attempt $i failed, retrying..."; sleep 5; }; \
|
|
31
|
+
done && \
|
|
27
32
|
rm -rf /var/lib/apt/lists/*
|
|
28
33
|
|
|
29
34
|
# Set Puppeteer environment variables
|