@link-assistant/web-capture 1.1.2 → 1.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/CHANGELOG.md +57 -13
  2. package/Dockerfile +26 -21
  3. package/README.md +290 -160
  4. package/bin/web-capture.js +622 -17
  5. package/package.json +15 -4
  6. package/src/animation.js +283 -0
  7. package/src/archive.js +225 -0
  8. package/src/batch.js +180 -0
  9. package/src/browser.js +150 -111
  10. package/src/docx.js +182 -0
  11. package/src/extract-images.js +120 -0
  12. package/src/figures.js +208 -0
  13. package/src/gdocs.js +335 -0
  14. package/src/image.js +65 -8
  15. package/src/index.js +15 -0
  16. package/src/latex.js +127 -0
  17. package/src/lib.js +281 -1
  18. package/src/localize-images.js +202 -0
  19. package/src/markdown.js +7 -2
  20. package/src/metadata.js +333 -0
  21. package/src/pdf.js +76 -0
  22. package/src/popups.js +171 -0
  23. package/src/postprocess.js +263 -0
  24. package/src/retry.js +48 -0
  25. package/src/themed-image.js +167 -0
  26. package/src/verify.js +312 -0
  27. package/tests/e2e/docker.test.js +51 -12
  28. package/tests/e2e/process.test.js +34 -10
  29. package/tests/fixtures/habr-articles.js +45 -0
  30. package/tests/integration/api-endpoints.test.js +231 -0
  31. package/tests/integration/habr-article.test.js +337 -0
  32. package/tests/mock/index.test.js +5 -3
  33. package/tests/unit/animation.test.js +33 -0
  34. package/tests/unit/batch.test.js +114 -0
  35. package/tests/unit/browser.test.js +38 -2
  36. package/tests/unit/extract-images.test.js +233 -0
  37. package/tests/unit/figures.test.js +74 -0
  38. package/tests/unit/gdocs.test.js +278 -0
  39. package/tests/unit/heading-image.test.js +49 -0
  40. package/tests/unit/latex.test.js +123 -0
  41. package/tests/unit/localize-images.test.js +78 -0
  42. package/tests/unit/metadata.test.js +140 -0
  43. package/tests/unit/postprocess.test.js +110 -0
  44. package/tests/unit/retry.test.js +111 -0
  45. package/tests/unit/verify.test.js +182 -0
  46. package/.github/workflows/release.yml +0 -294
  47. package/ARCHITECTURE.md +0 -982
  48. package/LICENSE +0 -24
  49. package/apply.sh +0 -166
  50. package/docs/market-analysis-2025-05-13.md +0 -64
  51. package/scripts/changeset-version.mjs +0 -38
  52. package/scripts/create-github-release.mjs +0 -93
  53. package/scripts/create-manual-changeset.mjs +0 -88
  54. package/scripts/format-github-release.mjs +0 -83
  55. package/scripts/format-release-notes.mjs +0 -218
  56. package/scripts/instant-version-bump.mjs +0 -121
  57. package/scripts/publish-to-npm.mjs +0 -128
  58. package/scripts/setup-npm.mjs +0 -37
  59. package/scripts/validate-changeset.mjs +0 -106
  60. package/scripts/version-and-commit.mjs +0 -237
package/CHANGELOG.md CHANGED
@@ -1,22 +1,66 @@
1
- # web-capture
1
+ # @link-assistant/web-capture
2
2
 
3
- ## 1.1.2
3
+ ## 1.7.1
4
4
 
5
5
  ### Patch Changes
6
6
 
7
- - a9ebb49: fix: configure scoped package name @link-assistant/web-capture for npm publishing
7
+ - 3ee464b: Add heading-image parity guard rail tests to prevent <img> inside headings from being silently dropped during HTML-to-Markdown conversion
8
8
 
9
- Updated package.json and all release scripts to use the correct scoped npm package name `@link-assistant/web-capture` instead of the incorrectly configured `my-package` and unscoped `web-capture` names.
9
+ ## 1.7.0
10
10
 
11
- ## 1.1.1
11
+ ### Minor Changes
12
+
13
+ - b6d77d6: Fix archive quality bugs: relative image paths in markdown, HTML pretty-printing, entity decoding, and rename article to document filenames
14
+
15
+ ## 1.5.1
16
+
17
+ ### Patch Changes
18
+
19
+ - e6dea94: Fix Google Docs capture: archive produces real zip, default markdown extracts images to files instead of stripping
20
+
21
+ ## 1.5.0
22
+
23
+ ### Minor Changes
24
+
25
+ - a463061: Markdown mode now keeps original remote image URLs and strips base64 by default (single-file output).
26
+ Archive mode downloads images to images/ folder by default (keepOriginalLinks=false, embedImages=false).
27
+ API and CLI share identical defaults for all modes.
28
+ Add --embed-images, --images-dir, --no-extract-images, --archive, --data-dir, --keep-original-links flags.
29
+ Add embedImages and keepOriginalLinks query params to /markdown and /archive API endpoints.
30
+ Change /markdown API defaults: keepOriginalLinks=true, embedImages=false.
31
+ Remove --enhanced umbrella flag; make sub-features default to true.
32
+ Change default format from html to markdown.
33
+ Auto-derive output directory from URL when -o is omitted.
34
+ Use content-hash filenames for extracted images instead of positional numbering.
35
+ Replace process.env ternaries with lino-arguments getenv() helper.
36
+ Add `WEB_CAPTURE_*` environment variable support for all feature flags.
37
+
38
+ ## 1.4.3
12
39
 
13
40
  ### Patch Changes
14
41
 
15
- - 155d3a6: Add comprehensive CI/CD pipeline based on js-ai-driven-development-pipeline-template
16
- - Add GitHub Actions workflow for automated testing, linting, and releases
17
- - Integrate changesets for version management and changelog generation
18
- - Add ESLint, Prettier, and JSCPD for code quality checks
19
- - Configure lint-staged and Husky for pre-commit hooks
20
- - Add e2e tests that work in CI with Playwright
21
- - Automated npm publishing via OIDC trusted publishing
22
- - GitHub release generation with formatted notes
42
+ - c9d1132: Fix cargo publish token resolution: fallback to CARGO_TOKEN when CARGO_REGISTRY_TOKEN is not set, and fail instead of silently skipping publish
43
+
44
+ ## 1.4.2
45
+
46
+ ### Patch Changes
47
+
48
+ - 94c1953: Update lino-arguments to ^0.3.0, fix npm publish OIDC fallback, and align CI/CD workflows with reference repo best practices
49
+
50
+ ## 1.4.1
51
+
52
+ ### Patch Changes
53
+
54
+ - 4570a2b: Fix CI/CD release pipeline: resolve git show path bug in version-and-commit.mjs where `git show origin/main:package.json` failed because git show uses repo-root-relative paths (should be `js/package.json`). Add npx-based fallback in setup-npm.mjs for Node.js 22.22.2 broken npm issue.
55
+
56
+ ## 1.4.0
57
+
58
+ ### Minor Changes
59
+
60
+ - f464988: Add Google Docs document to Markdown capture support with API token authentication
61
+
62
+ ## 1.3.0
63
+
64
+ ### Minor Changes
65
+
66
+ - 141458e: Implement all web capture best practices from meta-theory reference scripts (R1-R7): LaTeX formula extraction (Habr, KaTeX, MathJax), article metadata extraction, markdown post-processing pipeline, animation capture with loop detection, dual-theme screenshots, figure image extraction, markdown image localization, content verification with fuzzy matching, and batch processing with JSON/MJS configuration. Add enhanced HTML-to-markdown conversion with configurable options, 3 new API endpoints (/animation, /figures, /themed-image), and expanded CLI flags.
package/Dockerfile CHANGED
@@ -3,27 +3,32 @@ FROM node:22-bullseye
3
3
  WORKDIR /app
4
4
 
5
5
  # Install dependencies for headless Chrome (used by both Puppeteer and Playwright)
6
- RUN apt-get update && apt-get install -y \
7
- chromium \
8
- fonts-liberation \
9
- libasound2 \
10
- libatk-bridge2.0-0 \
11
- libatk1.0-0 \
12
- libatspi2.0-0 \
13
- libcups2 \
14
- libdbus-1-3 \
15
- libdrm2 \
16
- libgbm1 \
17
- libgtk-3-0 \
18
- libnspr4 \
19
- libnss3 \
20
- libxcomposite1 \
21
- libxdamage1 \
22
- libxfixes3 \
23
- libxrandr2 \
24
- libxshmfence1 \
25
- xdg-utils \
26
- --no-install-recommends && \
6
+ # Retry loop handles transient network errors from Debian mirrors in CI
7
+ RUN apt-get update && \
8
+ for i in 1 2 3; do \
9
+ apt-get install -y \
10
+ chromium \
11
+ fonts-liberation \
12
+ libasound2 \
13
+ libatk-bridge2.0-0 \
14
+ libatk1.0-0 \
15
+ libatspi2.0-0 \
16
+ libcups2 \
17
+ libdbus-1-3 \
18
+ libdrm2 \
19
+ libgbm1 \
20
+ libgtk-3-0 \
21
+ libnspr4 \
22
+ libnss3 \
23
+ libxcomposite1 \
24
+ libxdamage1 \
25
+ libxfixes3 \
26
+ libxrandr2 \
27
+ libxshmfence1 \
28
+ xdg-utils \
29
+ --no-install-recommends && break || \
30
+ { echo "apt-get install attempt $i failed, retrying..."; sleep 5; }; \
31
+ done && \
27
32
  rm -rf /var/lib/apt/lists/*
28
33
 
29
34
  # Set Puppeteer environment variables