@link-assistant/web-capture 1.1.2 → 1.4.2

package/CHANGELOG.md

@@ -1,22 +1,25 @@
-# web-capture
+# @link-assistant/web-capture
 
-## 1.1.2
+## 1.4.2
 
 ### Patch Changes
 
-- a9ebb49: fix: configure scoped package name @link-assistant/web-capture for npm publishing
+- 94c1953: Update lino-arguments to ^0.3.0, fix npm publish OIDC fallback, and align CI/CD workflows with reference repo best practices
 
-  Updated package.json and all release scripts to use the correct scoped npm package name `@link-assistant/web-capture` instead of the incorrectly configured `my-package` and unscoped `web-capture` names.
-
-## 1.1.1
+## 1.4.1
 
 ### Patch Changes
 
-- 155d3a6: Add comprehensive CI/CD pipeline based on js-ai-driven-development-pipeline-template
-  - Add GitHub Actions workflow for automated testing, linting, and releases
-  - Integrate changesets for version management and changelog generation
-  - Add ESLint, Prettier, and JSCPD for code quality checks
-  - Configure lint-staged and Husky for pre-commit hooks
-  - Add e2e tests that work in CI with Playwright
-  - Automated npm publishing via OIDC trusted publishing
-  - GitHub release generation with formatted notes
+- 4570a2b: Fix CI/CD release pipeline: resolve git show path bug in version-and-commit.mjs where `git show origin/main:package.json` failed because git show uses repo-root-relative paths (should be `js/package.json`). Add npx-based fallback in setup-npm.mjs for Node.js 22.22.2 broken npm issue.
+
+## 1.4.0
+
+### Minor Changes
+
+- f464988: Add Google Docs document to Markdown capture support with API token authentication
+
+## 1.3.0
+
+### Minor Changes
+
+- 141458e: Implement all web capture best practices from meta-theory reference scripts (R1-R7): LaTeX formula extraction (Habr, KaTeX, MathJax), article metadata extraction, markdown post-processing pipeline, animation capture with loop detection, dual-theme screenshots, figure image extraction, markdown image localization, content verification with fuzzy matching, and batch processing with JSON/MJS configuration. Add enhanced HTML-to-markdown conversion with configurable options, 3 new API endpoints (/animation, /figures, /themed-image), and expanded CLI flags.

package/README.md

@@ -1,30 +1,69 @@
-# web-capture
+# web-capture (JavaScript/Node.js)
 
-<img width="1824" alt="Screenshot 2025-05-12 at 3 49 32 AM" src="https://github.com/user-attachments/assets/cbf63dec-7dcd-40e7-9d5d-eddc49fe6169" />
+[![npm version](https://img.shields.io/npm/v/@link-assistant/web-capture?label=npm&color=blue)](https://www.npmjs.com/package/@link-assistant/web-capture)
+[![npm downloads](https://img.shields.io/npm/dm/@link-assistant/web-capture?label=downloads&color=blue)](https://www.npmjs.com/package/@link-assistant/web-capture)
+[![GitHub Release](https://img.shields.io/github/v/release/link-assistant/web-capture?label=GitHub%20Release)](https://github.com/link-assistant/web-capture/releases)
+[![CI - JavaScript](https://img.shields.io/github/actions/workflow/status/link-assistant/web-capture/js.yml?branch=main&label=CI)](https://github.com/link-assistant/web-capture/actions/workflows/js.yml)
+[![License: Unlicense](https://img.shields.io/badge/license-Unlicense-green)](https://unlicense.org)
 
 A CLI and microservice to fetch URLs and render them as:
 
 - **HTML**: Rendered page content
 - **Markdown**: Converted from HTML
-- **PNG screenshot**: Full page capture
+- **PNG/JPEG screenshot**: Viewport or full-page capture with theme support
+- **ZIP archive**: Markdown + locally downloaded images
+- **PDF**: Print-quality document with embedded images
+- **DOCX**: Word document with embedded images
+
+## Installation
+
+### From npm
+
+```bash
+npm install -g @link-assistant/web-capture
+```
+
+### From Source
+
+```bash
+cd js
+npm install
+# or
+yarn install
+```
 
 ## Quick Start
 
 ### CLI Usage
 
 ```bash
-# Install globally
-npm install -g web-capture
-
 # Capture a URL as HTML (output to stdout)
 web-capture https://example.com
 
 # Capture as Markdown and save to file
 web-capture https://example.com --format markdown --output page.md
 
-# Take a screenshot
+# Take a PNG screenshot
 web-capture https://example.com --format png --output screenshot.png
 
+# Take a JPEG screenshot with custom quality
+web-capture https://example.com --format jpeg --quality 60 --output screenshot.jpg
+
+# Dark theme full-page screenshot
+web-capture https://example.com --format png --theme dark --fullPage --output dark.png
+
+# Custom viewport size
+web-capture https://example.com --format png --width 1920 --height 1080 -o wide.png
+
+# Download as PDF
+web-capture https://example.com --format pdf --output page.pdf
+
+# Download as DOCX
+web-capture https://example.com --format docx --output page.docx
+
+# Download as ZIP archive with local images
+web-capture https://example.com --format archive --output site.zip
+
 # Start as API server
 web-capture --serve
 
@@ -32,200 +71,206 @@ web-capture --serve
 web-capture --serve --port 8080
 ```
 
-### API Endpoints (Server Mode)
+### Downloading Habr Articles
 
-- **HTML**: GET /html?url=<URL>
-- **Markdown**: GET /markdown?url=<URL>
-- **PNG screenshot**: GET /image?url=<URL>
-
-## Installation
+The service is tested against real Habr.com articles. Here are examples for each format:
 
 ```bash
-npm install
-# or
-yarn install
-```
+# Markdown (default: remote image links)
+web-capture https://habr.com/en/articles/895896/ -f markdown -o article.md
 
-## CLI Reference
+# ZIP archive with local images (markdown format, default)
+web-capture https://habr.com/en/articles/895896/ -f archive -o article.zip
 
-### Server Mode
+# ZIP archive with local images (HTML format with CSS)
+web-capture https://habr.com/en/articles/895896/ -f archive --documentFormat html -o article-html.zip
 
-Start the API server:
+# Light theme full-page PNG
+web-capture https://habr.com/en/articles/895896/ -f png --theme light --fullPage -o light.png
 
-```bash
-web-capture --serve [--port <port>]
-```
+# Dark theme full-page PNG
+web-capture https://habr.com/en/articles/895896/ -f png --theme dark --fullPage -o dark.png
 
-| Option    | Short | Description              | Default            |
-| --------- | ----- | ------------------------ | ------------------ |
-| `--serve` | `-s`  | Start as HTTP API server | -                  |
-| `--port`  | `-p`  | Port to listen on        | 3000 (or PORT env) |
-
-### Capture Mode
+# JPEG with 90% quality
+web-capture https://habr.com/en/articles/895896/ -f jpeg --quality 90 -o article.jpg
 
-Capture a URL directly:
+# PDF with dark theme
+web-capture https://habr.com/en/articles/895896/ -f pdf --theme dark -o article.pdf
 
-```bash
-web-capture <url> [options]
+# DOCX with embedded images
+web-capture https://habr.com/en/articles/895896/ -f docx -o article.docx
 ```
 
-| Option     | Short | Description                                           | Default                                  |
-| ---------- | ----- | ----------------------------------------------------- | ---------------------------------------- |
-| `--format` | `-f`  | Output format: `html`, `markdown`/`md`, `image`/`png` | `html`                                   |
-| `--output` | `-o`  | Output file path                                      | stdout (text) or auto-generated (images) |
-| `--engine` | `-e`  | Browser engine: `puppeteer`, `playwright`             | `puppeteer` (or BROWSER_ENGINE env)      |
-
-### Examples
+## API Endpoints (Server Mode)
 
-```bash
-# Capture HTML to stdout
-web-capture https://example.com
+Start the server with `web-capture --serve` and use the endpoints below.
 
-# Capture Markdown to file
-web-capture https://example.com -f markdown -o page.md
-
-# Take screenshot with Playwright engine
-web-capture https://example.com -f png -e playwright -o screenshot.png
+### HTML Endpoint
 
-# Pipe HTML to another command
-web-capture https://example.com | grep "title"
+```
+GET /html?url=<URL>&engine=<ENGINE>
 ```
 
-## Available Commands
+Returns the raw HTML content of the specified URL.
 
-### Development
+| Parameter | Required | Description                          | Default   |
+| --------- | -------- | ------------------------------------ | --------- |
+| `url`     | Yes      | URL to fetch                         | -         |
+| `engine`  | No       | Browser engine: puppeteer/playwright | puppeteer |
 
-- `yarn dev` - Start the development server with hot reloading using nodemon
-- `yarn start` - Start the service using Docker Compose
+### Markdown Endpoint
 
-### Testing
+```
+GET /markdown?url=<URL>
+```
 
-- `yarn test` - Run all unit tests
-- `yarn test:watch` - Run tests in watch mode
-- `yarn test:e2e` - Run end-to-end tests
-- `yarn test:e2e:docker` - Run end-to-end tests against Docker container
-- `yarn test:all` - Run all tests including build and e2e tests
+Converts the HTML content of the specified URL to Markdown format.
 
-### Building
+| Parameter | Required | Description  | Default |
+| --------- | -------- | ------------ | ------- |
+| `url`     | Yes      | URL to fetch | -       |
 
-- `yarn build` - Build and start the Docker container
+### Image Endpoint
 
-### Examples
+```
+GET /image?url=<URL>&format=png&theme=dark&width=1920&height=1080&fullPage=true
+```
 
-- `yarn examples:python` - Run Python example scripts
-- `yarn examples:javascript` - Run JavaScript example scripts
-- `yarn examples` - Run all examples (requires build)
+Returns a screenshot of the specified URL.
 
-## Usage
+| Parameter       | Required | Description                              | Default         |
+| --------------- | -------- | ---------------------------------------- | --------------- |
+| `url`           | Yes      | URL to capture                           | -               |
+| `engine`        | No       | Browser engine: puppeteer/playwright     | puppeteer       |
+| `format`        | No       | Image format: `png` (lossless) or `jpeg` | png             |
+| `quality`       | No       | JPEG quality 0-100 (ignored for PNG)     | 80              |
+| `width`         | No       | Viewport width in pixels                 | 1280            |
+| `height`        | No       | Viewport height in pixels                | 800             |
+| `fullPage`      | No       | Capture full scrollable page             | false           |
+| `theme`         | No       | Color scheme: `light`, `dark`            | browser default |
+| `dismissPopups` | No       | Auto-close cookie/consent popups         | true            |
 
-### Local Development
+### Archive Endpoint
 
-```bash
-yarn dev
-curl http://localhost:3000/html?url=https://example.com
+```
+GET /archive?url=<URL>&localImages=true&documentFormat=markdown
 ```
 
-### Docker
+Returns a ZIP archive containing either `article.md` or `article.html` and asset directories (`images/`, `css/`).
 
-```bash
-# Build and run using Docker Compose
-yarn start
+| Parameter        | Required | Description                              | Default    |
+| ---------------- | -------- | ---------------------------------------- | ---------- |
+| `url`            | Yes      | URL to archive                           | -          |
+| `localImages`    | No       | Download images locally into the archive | true       |
+| `documentFormat` | No       | Document format: `markdown` or `html`    | `markdown` |
 
-# Or manually
-docker build -t web-capture .
-docker run -p 3000:3000 web-capture
-```
+**Archive structure** (with `localImages=true`):
 
-## API Endpoints
+```
+archive.zip
+├── article.md        # or article.html when documentFormat=html
+├── images/
+│   ├── image-1.png
+│   ├── image-2.jpg
+│   └── ...
+└── css/              # only when documentFormat=html
+    ├── style-1.css
+    └── ...
+```
 
-### HTML Endpoint
+### PDF Endpoint
 
-```bash
-GET /html?url=<URL>&engine=<ENGINE>
+```
+GET /pdf?url=<URL>&theme=light
 ```
 
-Returns the raw HTML content of the specified URL.
+Returns a PDF document rendered by the browser engine (all images embedded).
 
-**Parameters:**
+| Parameter       | Required | Description                     | Default         |
+| --------------- | -------- | ------------------------------- | --------------- |
+| `url`           | Yes      | URL to convert                  | -               |
+| `engine`        | No       | Browser engine                  | puppeteer       |
+| `width`         | No       | Viewport width in pixels        | 1280            |
+| `height`        | No       | Viewport height in pixels       | 800             |
+| `theme`         | No       | Color scheme: `light`, `dark`   | browser default |
+| `dismissPopups` | No       | Auto-close popups before export | true            |
 
-- `url` (required): The URL to fetch
-- `engine` (optional): Browser engine to use (`puppeteer` or `playwright`). Default: `puppeteer`
+### DOCX Endpoint
 
-**Examples:**
+```
+GET /docx?url=<URL>
+```
 
-```bash
-# Using default Puppeteer engine
-curl http://localhost:3000/html?url=https://example.com
+Returns a DOCX document with embedded images.
 
-# Using Playwright engine
-curl http://localhost:3000/html?url=https://example.com&engine=playwright
-```
+| Parameter | Required | Description    | Default |
+| --------- | -------- | -------------- | ------- |
+| `url`     | Yes      | URL to convert | -       |
 
-### Markdown Endpoint
+### Fetch/Stream Endpoints
 
-```bash
-GET /markdown?url=<URL>
+```
+GET /fetch?url=<URL>
+GET /stream?url=<URL>
 ```
 
-Converts the HTML content of the specified URL to Markdown format.
+Proxy fetch and streaming proxy endpoints.
 
-### Image Endpoint
+## CLI Reference
+
+### Server Mode
 
 ```bash
-GET /image?url=<URL>&engine=<ENGINE>
+web-capture --serve [--port <port>]
 ```
 
-Returns a PNG screenshot of the specified URL.
-
-**Parameters:**
-
-- `url` (required): The URL to capture
-- `engine` (optional): Browser engine to use (`puppeteer` or `playwright`). Default: `puppeteer`
+| Option    | Short | Description              | Default            |
+| --------- | ----- | ------------------------ | ------------------ |
+| `--serve` | `-s`  | Start as HTTP API server | -                  |
+| `--port`  | `-p`  | Port to listen on        | 3000 (or PORT env) |
 
-**Examples:**
+### Capture Mode
 
 ```bash
-# Using default Puppeteer engine
-curl http://localhost:3000/image?url=https://example.com > screenshot.png
-
-# Using Playwright engine
-curl http://localhost:3000/image?url=https://example.com&engine=playwright > screenshot.png
+web-capture <url> [options]
 ```
 
+| Option             | Short | Description                                    | Default                                  |
+| ------------------ | ----- | ---------------------------------------------- | ---------------------------------------- |
+| `--format`         | `-f`  | Output format (see below)                      | `html`                                   |
+| `--output`         | `-o`  | Output file path                               | stdout (text) or auto-generated (images) |
+| `--engine`         | `-e`  | Browser engine: `puppeteer`, `playwright`      | `puppeteer` (or BROWSER_ENGINE env)      |
+| `--theme`          | `-t`  | Color scheme: `light`, `dark`, `no-preference` | browser default                          |
+| `--width`          |       | Viewport width in pixels                       | 1280                                     |
+| `--height`         |       | Viewport height in pixels                      | 800                                      |
+| `--quality`        |       | JPEG quality 0-100                             | 80                                       |
+| `--fullPage`       |       | Capture full scrollable page                   | false                                    |
+| `--localImages`    |       | Download images locally in archive mode        | true                                     |
+| `--documentFormat` |       | Document format in archive: `markdown`, `html` | `markdown`                               |
+
+**Supported formats:**
+
+| Format            | Description                              |
+| ----------------- | ---------------------------------------- |
+| `html`            | Rendered HTML (default)                  |
+| `markdown` / `md` | Markdown conversion                      |
+| `image` / `png`   | PNG screenshot (lossless)                |
+| `jpeg`            | JPEG screenshot (configurable quality)   |
+| `pdf`             | PDF with embedded images                 |
+| `docx`            | Word document with embedded images       |
+| `archive` / `zip` | ZIP archive with markdown + local images |
+
 ## Configuration
 
-web-capture uses [lino-arguments](https://github.com/link-foundation/lino-arguments) for unified configuration management. Configuration values are resolved with the following priority (highest to lowest):
+Configuration values are resolved with the following priority (highest to lowest):
 
 1. **CLI arguments**: `--port 8080`
 2. **Environment variables**: `PORT=8080`
-3. **Custom configuration file**: `--configuration path/to/custom.lenv`
-4. **Default .lenv file**: `.lenv` in the project root
-5. **Built-in defaults**
-
-### Configuration File (.lenv)
-
-Create a `.lenv` file in your project root using Links Notation format:
-
-```lenv
-# Server configuration
-PORT: 3000
-
-# Browser engine (puppeteer or playwright)
-BROWSER_ENGINE: puppeteer
-```
-
-### Using Custom Configuration Files
-
-Specify a custom configuration file path:
-
-```bash
-web-capture --serve --configuration /path/to/custom.lenv
-```
+3. **Default .lenv file**: `.lenv` in the project root
+4. **Built-in defaults**
 
 ### Environment Variables
 
-All configuration options support environment variables:
-
 ```bash
 # Set port via environment variable
 export PORT=8080
@@ -243,26 +288,91 @@ The service supports both **Puppeteer** and **Playwright** browser engines:
 - **Puppeteer**: Default engine, mature and well-tested
 - **Playwright**: Alternative engine with similar capabilities
 
-You can choose the engine using:
-
-- CLI argument: `--engine playwright`
-- Environment variable: `BROWSER_ENGINE=playwright`
-- Configuration file: `BROWSER_ENGINE: playwright` in `.lenv`
-
 **Supported engine values:**
 
 - `puppeteer` or `pptr` - Use Puppeteer
 - `playwright` or `pw` - Use Playwright
 
+## Popup/Modal Dismissal
+
+By default, the image, PDF, and archive endpoints automatically dismiss common popups before capture:
+
+- Google Funding Choices consent dialogs
+- Cookie consent banners
+- Generic modals and overlays
+- Fixed-position overlays covering content
+
+Set `dismissPopups=false` to disable this behavior.
+
+## Docker
+
+```bash
+# Build and run using Docker Compose
+docker compose up -d
+
+# Or manually
+docker build -t web-capture-js .
+docker run -p 3000:3000 web-capture-js
+```
+
 ## Development
 
-The service is built with:
+### Available Commands
+
+- `npm run dev` - Start the development server with hot reloading
+- `npm run start` - Start the service using Docker Compose
+- `npm test` - Run all unit tests
+- `npm run lint` - Check code with ESLint
+- `npm run format` - Format code with Prettier
+
+### Testing
+
+```bash
+npm test                    # Run unit and integration tests
+npm run test:e2e            # Run end-to-end tests
+npm run test:all            # Run all tests including build
+```
+
+## Built With
 
 - Express.js for the web server
-- Puppeteer and Playwright for headless browser automation and screenshots
+- Puppeteer and Playwright for headless browser automation
 - Turndown for HTML to Markdown conversion
+- archiver for ZIP creation
+- docx for DOCX generation
 - Jest for testing
 
+## Library Usage
+
+You can also use web-capture as a Node.js library:
+
+```javascript
+import {
+  fetchHtml,
+  convertHtmlToMarkdown,
+} from '@link-assistant/web-capture/src/lib.js';
+import { createBrowser } from '@link-assistant/web-capture/src/browser.js';
+import { dismissPopups } from '@link-assistant/web-capture/src/popups.js';
+
+// Fetch and convert to markdown
+const html = await fetchHtml('https://habr.com/en/articles/895896/');
+const markdown = convertHtmlToMarkdown(
+  html,
+  'https://habr.com/en/articles/895896/'
+);
+
+// Take a themed screenshot
+const browser = await createBrowser('playwright', { colorScheme: 'dark' });
+const page = await browser.newPage();
+await page.setViewport({ width: 1920, height: 1080 });
+await page.goto('https://habr.com/en/articles/895896/', {
+  waitUntil: 'networkidle0',
+});
+await dismissPopups(page);
+const buffer = await page.screenshot({ type: 'png', fullPage: true });
+await browser.close();
+```
+
 ## License
 
-UNLICENSED
+[Unlicense](../LICENSE) — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See [https://unlicense.org](https://unlicense.org) for details.