@link-assistant/web-capture 1.1.2 → 1.7.1

package/README.md

@@ -1,241 +1,306 @@
-# web-capture
+# web-capture (JavaScript/Node.js)
 
-<img width="1824" alt="Screenshot 2025-05-12 at 3 49 32 AM" src="https://github.com/user-attachments/assets/cbf63dec-7dcd-40e7-9d5d-eddc49fe6169" />
+[![npm version](https://img.shields.io/npm/v/@link-assistant/web-capture?label=npm&color=blue)](https://www.npmjs.com/package/@link-assistant/web-capture)
+[![npm downloads](https://img.shields.io/npm/dm/@link-assistant/web-capture?label=downloads&color=blue)](https://www.npmjs.com/package/@link-assistant/web-capture)
+[![GitHub Release](https://img.shields.io/github/v/release/link-assistant/web-capture?label=GitHub%20Release)](https://github.com/link-assistant/web-capture/releases)
+[![CI - JavaScript](https://img.shields.io/github/actions/workflow/status/link-assistant/web-capture/js.yml?branch=main&label=CI)](https://github.com/link-assistant/web-capture/actions/workflows/js.yml)
+[![License: Unlicense](https://img.shields.io/badge/license-Unlicense-green)](https://unlicense.org)
 
 A CLI and microservice to fetch URLs and render them as:
 
+- **Markdown**: Converted from HTML with image extraction (default)
 - **HTML**: Rendered page content
-- **Markdown**: Converted from HTML
-- **PNG screenshot**: Full page capture
-
-## Quick Start
-
-### CLI Usage
-
-```bash
-# Install globally
-npm install -g web-capture
-
-# Capture a URL as HTML (output to stdout)
-web-capture https://example.com
-
-# Capture as Markdown and save to file
-web-capture https://example.com --format markdown --output page.md
-
-# Take a screenshot
-web-capture https://example.com --format png --output screenshot.png
-
-# Start as API server
-web-capture --serve
-
-# Start server on custom port
-web-capture --serve --port 8080
-```
-
-### API Endpoints (Server Mode)
-
-- **HTML**: GET /html?url=<URL>
-- **Markdown**: GET /markdown?url=<URL>
-- **PNG screenshot**: GET /image?url=<URL>
+- **PNG/JPEG screenshot**: Viewport or full-page capture with theme support
+- **ZIP archive**: Markdown + locally downloaded images
+- **PDF**: Print-quality document with embedded images
+- **DOCX**: Word document with embedded images
 
 ## Installation
 
-```bash
-npm install
-# or
-yarn install
-```
-
-## CLI Reference
-
-### Server Mode
-
-Start the API server:
+### From npm
 
 ```bash
-web-capture --serve [--port <port>]
+npm install -g @link-assistant/web-capture
 ```
 
-| Option    | Short | Description              | Default            |
-| --------- | ----- | ------------------------ | ------------------ |
-| `--serve` | `-s`  | Start as HTTP API server | -                  |
-| `--port`  | `-p`  | Port to listen on        | 3000 (or PORT env) |
-
-### Capture Mode
-
-Capture a URL directly:
+### From Source
 
 ```bash
-web-capture <url> [options]
+cd js
+npm install
 ```
 
-| Option     | Short | Description                                           | Default                                  |
-| ---------- | ----- | ----------------------------------------------------- | ---------------------------------------- |
-| `--format` | `-f`  | Output format: `html`, `markdown`/`md`, `image`/`png` | `html`                                   |
-| `--output` | `-o`  | Output file path                                      | stdout (text) or auto-generated (images) |
-| `--engine` | `-e`  | Browser engine: `puppeteer`, `playwright`             | `puppeteer` (or BROWSER_ENGINE env)      |
+## Quick Start
 
-### Examples
+### CLI Usage
 
 ```bash
-# Capture HTML to stdout
+# Capture a URL as Markdown (default format)
+# Output auto-derived to ./data/web-capture/<host>/<path>/document.md
 web-capture https://example.com
 
-# Capture Markdown to file
-web-capture https://example.com -f markdown -o page.md
+# Capture as Markdown to a specific file
+web-capture https://example.com -o page.md
 
-# Take screenshot with Playwright engine
-web-capture https://example.com -f png -e playwright -o screenshot.png
+# Write to stdout explicitly
+web-capture https://example.com -o -
 
-# Pipe HTML to another command
-web-capture https://example.com | grep "title"
-```
+# Capture as HTML
+web-capture https://example.com --format html -o page.html
 
-## Available Commands
+# Take a PNG screenshot
+web-capture https://example.com --format png -o screenshot.png
 
-### Development
+# Take a JPEG screenshot with custom quality
+web-capture https://example.com --format jpeg --quality 60 -o screenshot.jpg
 
-- `yarn dev` - Start the development server with hot reloading using nodemon
-- `yarn start` - Start the service using Docker Compose
+# Dark theme full-page screenshot
+web-capture https://example.com --format png --theme dark --fullPage -o dark.png
 
-### Testing
+# Custom viewport size
+web-capture https://example.com --format png --width 1920 --height 1080 -o wide.png
 
-- `yarn test` - Run all unit tests
-- `yarn test:watch` - Run tests in watch mode
-- `yarn test:e2e` - Run end-to-end tests
-- `yarn test:e2e:docker` - Run end-to-end tests against Docker container
-- `yarn test:all` - Run all tests including build and e2e tests
+# Download as PDF
+web-capture https://example.com --format pdf -o page.pdf
 
-### Building
+# Download as DOCX
+web-capture https://example.com --format docx -o page.docx
 
-- `yarn build` - Build and start the Docker container
+# Create a ZIP archive (default: zip format)
+web-capture https://example.com --archive
 
-### Examples
+# Create a ZIP archive with specific format
+web-capture https://example.com --archive zip -o site.zip
+web-capture https://example.com --archive tar.gz -o site.tar.gz
 
-- `yarn examples:python` - Run Python example scripts
-- `yarn examples:javascript` - Run JavaScript example scripts
-- `yarn examples` - Run all examples (requires build)
+# Keep images inline as base64 (opt-in)
+web-capture https://example.com --embed-images -o page.md
 
-## Usage
+# Custom images directory
+web-capture https://example.com --images-dir assets -o page.md
 
-### Local Development
+# Disable specific features
+web-capture https://example.com --no-extract-latex --no-post-process -o page.md
 
-```bash
-yarn dev
-curl http://localhost:3000/html?url=https://example.com
+# Start as API server
+web-capture --serve
+
+# Start server on custom port
+web-capture --serve --port 8080
 ```
 
-### Docker
+### Google Docs
 
 ```bash
-# Build and run using Docker Compose
-yarn start
+# Capture public Google Doc as Markdown
+web-capture https://docs.google.com/document/d/DOC_ID/edit
 
-# Or manually
-docker build -t web-capture .
-docker run -p 3000:3000 web-capture
+# Capture private Google Doc with API token
+web-capture https://docs.google.com/document/d/DOC_ID/edit --apiToken YOUR_TOKEN
 ```
 
-## API Endpoints
+## API Endpoints (Server Mode)
+
+Start the server with `web-capture --serve` and use the endpoints below.
 
 ### HTML Endpoint
 
-```bash
+```
 GET /html?url=<URL>&engine=<ENGINE>
 ```
 
 Returns the raw HTML content of the specified URL.
 
-**Parameters:**
+| Parameter | Required | Description                          | Default   |
+| --------- | -------- | ------------------------------------ | --------- |
+| `url`     | Yes      | URL to fetch                         | -         |
+| `engine`  | No       | Browser engine: puppeteer/playwright | puppeteer |
 
-- `url` (required): The URL to fetch
-- `engine` (optional): Browser engine to use (`puppeteer` or `playwright`). Default: `puppeteer`
+### Markdown Endpoint
 
-**Examples:**
+```
+GET /markdown?url=<URL>
+```
 
-```bash
-# Using default Puppeteer engine
-curl http://localhost:3000/html?url=https://example.com
+Converts the HTML content of the specified URL to Markdown format. By default, original remote image URLs are preserved and base64 data URIs are stripped (clean single-file output). Use `keepOriginalLinks=false` to strip all images, or `embedImages=true` to keep base64 images inline.
 
-# Using Playwright engine
-curl http://localhost:3000/html?url=https://example.com&engine=playwright
-```
+| Parameter           | Required | Description                                | Default |
+| ------------------- | -------- | ------------------------------------------ | ------- |
+| `url`               | Yes      | URL to fetch                               | -       |
+| `embedImages`       | No       | Keep base64 images inline (`true`/`false`) | `false` |
+| `keepOriginalLinks` | No       | Keep original remote URLs, strip base64    | `true`  |
 
-### Markdown Endpoint
+### Image Endpoint
 
-```bash
-GET /markdown?url=<URL>
+```
+GET /image?url=<URL>&format=png&theme=dark&width=1920&height=1080&fullPage=true
 ```
 
-Converts the HTML content of the specified URL to Markdown format.
+Returns a screenshot of the specified URL.
 
-### Image Endpoint
+| Parameter       | Required | Description                              | Default         |
+| --------------- | -------- | ---------------------------------------- | --------------- |
+| `url`           | Yes      | URL to capture                           | -               |
+| `engine`        | No       | Browser engine: puppeteer/playwright     | puppeteer       |
+| `format`        | No       | Image format: `png` (lossless) or `jpeg` | png             |
+| `quality`       | No       | JPEG quality 0-100 (ignored for PNG)     | 80              |
+| `width`         | No       | Viewport width in pixels                 | 1280            |
+| `height`        | No       | Viewport height in pixels                | 800             |
+| `fullPage`      | No       | Capture full scrollable page             | false           |
+| `theme`         | No       | Color scheme: `light`, `dark`            | browser default |
+| `dismissPopups` | No       | Auto-close cookie/consent popups         | true            |
+
+### Archive Endpoint
 
-```bash
-GET /image?url=<URL>&engine=<ENGINE>
+```
+GET /archive?url=<URL>&localImages=true&documentFormat=markdown
 ```
 
-Returns a PNG screenshot of the specified URL.
+Returns a ZIP archive containing either `document.md` or `document.html` and asset directories (`images/`, `css/`).
 
-**Parameters:**
+| Parameter           | Required | Description                                  | Default    |
+| ------------------- | -------- | -------------------------------------------- | ---------- |
+| `url`               | Yes      | URL to archive                               | -          |
+| `localImages`       | No       | Download images locally into the archive     | `true`     |
+| `documentFormat`    | No       | Document format: `markdown` or `html`        | `markdown` |
+| `embedImages`       | No       | Keep base64 images inline in the document    | `false`    |
+| `keepOriginalLinks` | No       | Keep original remote URLs (skip downloading) | `false`    |
 
-- `url` (required): The URL to capture
-- `engine` (optional): Browser engine to use (`puppeteer` or `playwright`). Default: `puppeteer`
+**Archive structure** (with `localImages=true`):
 
-**Examples:**
+```
+archive.zip
+├── document.md       # or document.html when documentFormat=html
+├── images/
+│   ├── image-1.png
+│   ├── image-2.jpg
+│   └── ...
+└── css/              # only when documentFormat=html
+    ├── style-1.css
+    └── ...
+```
 
-```bash
-# Using default Puppeteer engine
-curl http://localhost:3000/image?url=https://example.com > screenshot.png
+### PDF Endpoint
 
-# Using Playwright engine
-curl http://localhost:3000/image?url=https://example.com&engine=playwright > screenshot.png
+```
+GET /pdf?url=<URL>&theme=light
 ```
 
-## Configuration
+Returns a PDF document rendered by the browser engine (all images embedded).
 
-web-capture uses [lino-arguments](https://github.com/link-foundation/lino-arguments) for unified configuration management. Configuration values are resolved with the following priority (highest to lowest):
+| Parameter       | Required | Description                     | Default         |
+| --------------- | -------- | ------------------------------- | --------------- |
+| `url`           | Yes      | URL to convert                  | -               |
+| `engine`        | No       | Browser engine                  | puppeteer       |
+| `width`         | No       | Viewport width in pixels        | 1280            |
+| `height`        | No       | Viewport height in pixels       | 800             |
+| `theme`         | No       | Color scheme: `light`, `dark`   | browser default |
+| `dismissPopups` | No       | Auto-close popups before export | true            |
 
-1. **CLI arguments**: `--port 8080`
-2. **Environment variables**: `PORT=8080`
-3. **Custom configuration file**: `--configuration path/to/custom.lenv`
-4. **Default .lenv file**: `.lenv` in the project root
-5. **Built-in defaults**
+### DOCX Endpoint
 
-### Configuration File (.lenv)
+```
+GET /docx?url=<URL>
+```
 
-Create a `.lenv` file in your project root using Links Notation format:
+Returns a DOCX document with embedded images.
 
-```lenv
-# Server configuration
-PORT: 3000
+| Parameter | Required | Description    | Default |
+| --------- | -------- | -------------- | ------- |
+| `url`     | Yes      | URL to convert | -       |
 
-# Browser engine (puppeteer or playwright)
-BROWSER_ENGINE: puppeteer
+### Fetch/Stream Endpoints
+
+```
+GET /fetch?url=<URL>
+GET /stream?url=<URL>
 ```
 
-### Using Custom Configuration Files
+Proxy fetch and streaming proxy endpoints.
 
-Specify a custom configuration file path:
+## CLI Reference
+
+### Server Mode
 
 ```bash
-web-capture --serve --configuration /path/to/custom.lenv
+web-capture --serve [--port <port>]
 ```
 
-### Environment Variables
+| Option    | Short | Description              | Default            |
+| --------- | ----- | ------------------------ | ------------------ |
+| `--serve` | `-s`  | Start as HTTP API server | -                  |
+| `--port`  | `-p`  | Port to listen on        | 3000 (or PORT env) |
 
-All configuration options support environment variables:
+### Capture Mode
 
 ```bash
-# Set port via environment variable
-export PORT=8080
-web-capture --serve
-
-# Set browser engine
-export BROWSER_ENGINE=playwright
-web-capture https://example.com --format png
+web-capture <url> [options]
 ```
 
+| Option                      | Short | Description                                    | Default                             |
+| --------------------------- | ----- | ---------------------------------------------- | ----------------------------------- |
+| `--format`                  | `-f`  | Output format (see below)                      | `markdown`                          |
+| `--output`                  | `-o`  | Output file path. Use `-o -` for stdout        | auto-derived from URL               |
+| `--data-dir`                |       | Base directory for auto-derived output paths   | `./data/web-capture`                |
+| `--engine`                  | `-e`  | Browser engine: `puppeteer`, `playwright`      | `puppeteer` (or BROWSER_ENGINE env) |
+| `--theme`                   | `-t`  | Color scheme: `light`, `dark`, `no-preference` | browser default                     |
+| `--width`                   |       | Viewport width in pixels                       | 1280                                |
+| `--height`                  |       | Viewport height in pixels                      | 800                                 |
+| `--quality`                 |       | JPEG quality 0-100                             | 80                                  |
+| `--fullPage`                |       | Capture full scrollable page                   | false                               |
+| `--embed-images`            |       | Keep images as inline base64 data URIs         | false                               |
+| `--no-extract-images`       |       | Alias for `--embed-images`                     | false                               |
+| `--keep-original-links`     |       | Keep original remote URLs, strip base64        | false                               |
+| `--images-dir`              |       | Subdirectory name for extracted images         | `images`                            |
+| `--archive`                 |       | Create archive: `zip`, `7z`, `tar.gz`, `tar`   | -                                   |
+| `--document-format`         |       | Document format in archive: `markdown`, `html` | `markdown`                          |
+| `--localImages`             |       | Download images locally in archive mode        | true                                |
+| `--extract-latex`           |       | Extract LaTeX formulas                         | true                                |
+| `--no-extract-latex`        |       | Disable LaTeX extraction                       | -                                   |
+| `--extract-metadata`        |       | Extract article metadata                       | true                                |
+| `--no-extract-metadata`     |       | Disable metadata extraction                    | -                                   |
+| `--post-process`            |       | Apply post-processing                          | true                                |
+| `--no-post-process`         |       | Disable post-processing                        | -                                   |
+| `--detect-code-language`    |       | Detect code block languages                    | true                                |
+| `--no-detect-code-language` |       | Disable code language detection                | -                                   |
+
+**Supported formats:**
+
+| Format            | Description                              |
+| ----------------- | ---------------------------------------- |
+| `markdown` / `md` | Markdown conversion (default)            |
+| `html`            | Rendered HTML                            |
+| `image` / `png`   | PNG screenshot (lossless)                |
+| `jpeg`            | JPEG screenshot (configurable quality)   |
+| `pdf`             | PDF with embedded images                 |
+| `docx`            | Word document with embedded images       |
+| `archive` / `zip` | ZIP archive with markdown + local images |
+
+## Configuration
+
+Configuration values are resolved with the following priority (highest to lowest):
+
+1. **CLI arguments**: `--port 8080`
+2. **Environment variables**: `PORT=8080`
+3. **Default .lenv file**: `.lenv` in the project root
+4. **Built-in defaults**
+
+### Environment Variables
+
+| Variable                           | Description                         | Default              |
+| ---------------------------------- | ----------------------------------- | -------------------- |
+| `PORT`                             | Server port                         | `3000`               |
+| `BROWSER_ENGINE`                   | Browser engine                      | `puppeteer`          |
+| `API_TOKEN`                        | API token for authenticated capture | -                    |
+| `WEB_CAPTURE_DATA_DIR`             | Base directory for output           | `./data/web-capture` |
+| `WEB_CAPTURE_EMBED_IMAGES`         | `0`/`1` — keep images inline        | `0`                  |
+| `WEB_CAPTURE_KEEP_ORIGINAL_LINKS`  | `0`/`1` — keep original remote URLs | `0`                  |
+| `WEB_CAPTURE_IMAGES_DIR`           | Subdirectory for extracted images   | `images`             |
+| `WEB_CAPTURE_EXTRACT_LATEX`        | `0`/`1` — extract LaTeX             | `1`                  |
+| `WEB_CAPTURE_EXTRACT_METADATA`     | `0`/`1` — extract metadata          | `1`                  |
+| `WEB_CAPTURE_POST_PROCESS`         | `0`/`1` — post-processing           | `1`                  |
+| `WEB_CAPTURE_DETECT_CODE_LANGUAGE` | `0`/`1` — detect code langs         | `1`                  |
+
 ## Browser Engine Support
 
 The service supports both **Puppeteer** and **Playwright** browser engines:
@@ -243,26 +308,91 @@ The service supports both **Puppeteer** and **Playwright** browser engines:
 - **Puppeteer**: Default engine, mature and well-tested
 - **Playwright**: Alternative engine with similar capabilities
 
-You can choose the engine using:
-
-- CLI argument: `--engine playwright`
-- Environment variable: `BROWSER_ENGINE=playwright`
-- Configuration file: `BROWSER_ENGINE: playwright` in `.lenv`
-
 **Supported engine values:**
 
 - `puppeteer` or `pptr` - Use Puppeteer
 - `playwright` or `pw` - Use Playwright
 
+## Popup/Modal Dismissal
+
+By default, the image, PDF, and archive endpoints automatically dismiss common popups before capture:
+
+- Google Funding Choices consent dialogs
+- Cookie consent banners
+- Generic modals and overlays
+- Fixed-position overlays covering content
+
+Set `dismissPopups=false` to disable this behavior.
+
+## Docker
+
+```bash
+# Build and run using Docker Compose
+docker compose up -d
+
+# Or manually
+docker build -t web-capture-js .
+docker run -p 3000:3000 web-capture-js
+```
+
 ## Development
 
-The service is built with:
+### Available Commands
+
+- `npm run dev` - Start the development server with hot reloading
+- `npm run start` - Start the service using Docker Compose
+- `npm test` - Run all unit tests
+- `npm run lint` - Check code with ESLint
+- `npm run format` - Format code with Prettier
+
+### Testing
+
+```bash
+npm test                    # Run unit and integration tests
+npm run test:e2e            # Run end-to-end tests
+npm run test:all            # Run all tests including build
+```
+
+## Built With
 
 - Express.js for the web server
-- Puppeteer and Playwright for headless browser automation and screenshots
+- Puppeteer and Playwright for headless browser automation
 - Turndown for HTML to Markdown conversion
+- archiver for ZIP creation
+- docx for DOCX generation
 - Jest for testing
 
+## Library Usage
+
+You can also use web-capture as a Node.js library:
+
+```javascript
+import {
+  fetchHtml,
+  convertHtmlToMarkdown,
+} from '@link-assistant/web-capture/src/lib.js';
+import { createBrowser } from '@link-assistant/web-capture/src/browser.js';
+import { dismissPopups } from '@link-assistant/web-capture/src/popups.js';
+
+// Fetch and convert to markdown
+const html = await fetchHtml('https://habr.com/en/articles/895896/');
+const markdown = convertHtmlToMarkdown(
+  html,
+  'https://habr.com/en/articles/895896/'
+);
+
+// Take a themed screenshot
+const browser = await createBrowser('playwright', { colorScheme: 'dark' });
+const page = await browser.newPage();
+await page.setViewport({ width: 1920, height: 1080 });
+await page.goto('https://habr.com/en/articles/895896/', {
+  waitUntil: 'networkidle0',
+});
+await dismissPopups(page);
+const buffer = await page.screenshot({ type: 'png', fullPage: true });
+await browser.close();
+```
+
 ## License
 
-UNLICENSED
+[Unlicense](../LICENSE) — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See [https://unlicense.org](https://unlicense.org) for details.