@link-assistant/web-capture 1.1.2

package/CHANGELOG.md

@@ -0,0 +1,22 @@
+# web-capture
+
+## 1.1.2
+
+### Patch Changes
+
+- a9ebb49: fix: configure scoped package name @link-assistant/web-capture for npm publishing
+
+  Updated package.json and all release scripts to use the correct scoped npm package name `@link-assistant/web-capture` instead of the incorrectly configured `my-package` and unscoped `web-capture` names.
+
+## 1.1.1
+
+### Patch Changes
+
+- 155d3a6: Add comprehensive CI/CD pipeline based on js-ai-driven-development-pipeline-template
+  - Add GitHub Actions workflow for automated testing, linting, and releases
+  - Integrate changesets for version management and changelog generation
+  - Add ESLint, Prettier, and JSCPD for code quality checks
+  - Configure lint-staged and Husky for pre-commit hooks
+  - Add e2e tests that work in CI with Playwright
+  - Automated npm publishing via OIDC trusted publishing
+  - GitHub release generation with formatted notes

package/Dockerfile

@@ -0,0 +1,45 @@
+FROM node:22-bullseye
+
+WORKDIR /app
+
+# Install dependencies for headless Chrome (used by both Puppeteer and Playwright)
+RUN apt-get update && apt-get install -y \
+    chromium \
+    fonts-liberation \
+    libasound2 \
+    libatk-bridge2.0-0 \
+    libatk1.0-0 \
+    libatspi2.0-0 \
+    libcups2 \
+    libdbus-1-3 \
+    libdrm2 \
+    libgbm1 \
+    libgtk-3-0 \
+    libnspr4 \
+    libnss3 \
+    libxcomposite1 \
+    libxdamage1 \
+    libxfixes3 \
+    libxrandr2 \
+    libxshmfence1 \
+    xdg-utils \
+    --no-install-recommends && \
+    rm -rf /var/lib/apt/lists/*
+
+# Set Puppeteer environment variables
+ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
+ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
+ENV PUPPETEER_ARGS="--no-sandbox --disable-setuid-sandbox --disable-dev-shm-usage"
+
+# Set Playwright environment variables
+ENV PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
+ENV PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH=/usr/bin/chromium
+
+COPY package.json yarn.lock ./
+RUN yarn install --frozen-lockfile
+
+COPY . .
+
+EXPOSE 3000
+
+ENTRYPOINT ["node", "bin/web-capture.js", "--serve"]

package/LICENSE

@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <https://unlicense.org>

package/README.md

@@ -0,0 +1,268 @@
+# web-capture
+
+<img width="1824" alt="Screenshot 2025-05-12 at 3 49 32 AM" src="https://github.com/user-attachments/assets/cbf63dec-7dcd-40e7-9d5d-eddc49fe6169" />
+
+A CLI and microservice to fetch URLs and render them as:
+
+- **HTML**: Rendered page content
+- **Markdown**: Converted from HTML
+- **PNG screenshot**: Full page capture
+
+## Quick Start
+
+### CLI Usage
+
+```bash
+# Install globally
+npm install -g web-capture
+
+# Capture a URL as HTML (output to stdout)
+web-capture https://example.com
+
+# Capture as Markdown and save to file
+web-capture https://example.com --format markdown --output page.md
+
+# Take a screenshot
+web-capture https://example.com --format png --output screenshot.png
+
+# Start as API server
+web-capture --serve
+
+# Start server on custom port
+web-capture --serve --port 8080
+```
+
+### API Endpoints (Server Mode)
+
+- **HTML**: GET /html?url=<URL>
+- **Markdown**: GET /markdown?url=<URL>
+- **PNG screenshot**: GET /image?url=<URL>
+
+## Installation
+
+```bash
+npm install
+# or
+yarn install
+```
+
+## CLI Reference
+
+### Server Mode
+
+Start the API server:
+
+```bash
+web-capture --serve [--port <port>]
+```
+
+| Option    | Short | Description              | Default            |
+| --------- | ----- | ------------------------ | ------------------ |
+| `--serve` | `-s`  | Start as HTTP API server | -                  |
+| `--port`  | `-p`  | Port to listen on        | 3000 (or PORT env) |
+
+### Capture Mode
+
+Capture a URL directly:
+
+```bash
+web-capture <url> [options]
+```
+
+| Option     | Short | Description                                           | Default                                  |
+| ---------- | ----- | ----------------------------------------------------- | ---------------------------------------- |
+| `--format` | `-f`  | Output format: `html`, `markdown`/`md`, `image`/`png` | `html`                                   |
+| `--output` | `-o`  | Output file path                                      | stdout (text) or auto-generated (images) |
+| `--engine` | `-e`  | Browser engine: `puppeteer`, `playwright`             | `puppeteer` (or BROWSER_ENGINE env)      |
+
+### Examples
+
+```bash
+# Capture HTML to stdout
+web-capture https://example.com
+
+# Capture Markdown to file
+web-capture https://example.com -f markdown -o page.md
+
+# Take screenshot with Playwright engine
+web-capture https://example.com -f png -e playwright -o screenshot.png
+
+# Pipe HTML to another command
+web-capture https://example.com | grep "title"
+```
+
+## Available Commands
+
+### Development
+
+- `yarn dev` - Start the development server with hot reloading using nodemon
+- `yarn start` - Start the service using Docker Compose
+
+### Testing
+
+- `yarn test` - Run all unit tests
+- `yarn test:watch` - Run tests in watch mode
+- `yarn test:e2e` - Run end-to-end tests
+- `yarn test:e2e:docker` - Run end-to-end tests against Docker container
+- `yarn test:all` - Run all tests including build and e2e tests
+
+### Building
+
+- `yarn build` - Build and start the Docker container
+
+### Examples
+
+- `yarn examples:python` - Run Python example scripts
+- `yarn examples:javascript` - Run JavaScript example scripts
+- `yarn examples` - Run all examples (requires build)
+
+## Usage
+
+### Local Development
+
+```bash
+yarn dev
+curl http://localhost:3000/html?url=https://example.com
+```
+
+### Docker
+
+```bash
+# Build and run using Docker Compose
+yarn start
+
+# Or manually
+docker build -t web-capture .
+docker run -p 3000:3000 web-capture
+```
+
+## API Endpoints
+
+### HTML Endpoint
+
+```bash
+GET /html?url=<URL>&engine=<ENGINE>
+```
+
+Returns the raw HTML content of the specified URL.
+
+**Parameters:**
+
+- `url` (required): The URL to fetch
+- `engine` (optional): Browser engine to use (`puppeteer` or `playwright`). Default: `puppeteer`
+
+**Examples:**
+
+```bash
+# Using default Puppeteer engine
+curl http://localhost:3000/html?url=https://example.com
+
+# Using Playwright engine
+curl http://localhost:3000/html?url=https://example.com&engine=playwright
+```
+
+### Markdown Endpoint
+
+```bash
+GET /markdown?url=<URL>
+```
+
+Converts the HTML content of the specified URL to Markdown format.
+
+### Image Endpoint
+
+```bash
+GET /image?url=<URL>&engine=<ENGINE>
+```
+
+Returns a PNG screenshot of the specified URL.
+
+**Parameters:**
+
+- `url` (required): The URL to capture
+- `engine` (optional): Browser engine to use (`puppeteer` or `playwright`). Default: `puppeteer`
+
+**Examples:**
+
+```bash
+# Using default Puppeteer engine
+curl http://localhost:3000/image?url=https://example.com > screenshot.png
+
+# Using Playwright engine
+curl http://localhost:3000/image?url=https://example.com&engine=playwright > screenshot.png
+```
+
+## Configuration
+
+web-capture uses [lino-arguments](https://github.com/link-foundation/lino-arguments) for unified configuration management. Configuration values are resolved with the following priority (highest to lowest):
+
+1. **CLI arguments**: `--port 8080`
+2. **Environment variables**: `PORT=8080`
+3. **Custom configuration file**: `--configuration path/to/custom.lenv`
+4. **Default .lenv file**: `.lenv` in the project root
+5. **Built-in defaults**
+
+### Configuration File (.lenv)
+
+Create a `.lenv` file in your project root using Links Notation format:
+
+```lenv
+# Server configuration
+PORT: 3000
+
+# Browser engine (puppeteer or playwright)
+BROWSER_ENGINE: puppeteer
+```
+
+### Using Custom Configuration Files
+
+Specify a custom configuration file path:
+
+```bash
+web-capture --serve --configuration /path/to/custom.lenv
+```
+
+### Environment Variables
+
+All configuration options support environment variables:
+
+```bash
+# Set port via environment variable
+export PORT=8080
+web-capture --serve
+
+# Set browser engine
+export BROWSER_ENGINE=playwright
+web-capture https://example.com --format png
+```
+
+## Browser Engine Support
+
+The service supports both **Puppeteer** and **Playwright** browser engines:
+
+- **Puppeteer**: Default engine, mature and well-tested
+- **Playwright**: Alternative engine with similar capabilities
+
+You can choose the engine using:
+
+- CLI argument: `--engine playwright`
+- Environment variable: `BROWSER_ENGINE=playwright`
+- Configuration file: `BROWSER_ENGINE: playwright` in `.lenv`
+
+**Supported engine values:**
+
+- `puppeteer` or `pptr` - Use Puppeteer
+- `playwright` or `pw` - Use Playwright
+
+## Development
+
+The service is built with:
+
+- Express.js for the web server
+- Puppeteer and Playwright for headless browser automation and screenshots
+- Turndown for HTML to Markdown conversion
+- Jest for testing
+
+## License
+
+UNLICENSED

package/apply.sh

@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Usage: ./init.sh [directory]
+DIR=${1:-web-capture}
+echo "Initializing web-capture microservice in '$DIR'..."
+
+# Create project directory and enter
+mkdir -p "$DIR"
+cd "$DIR"
+
+# Initialize git
+git init
+
+echo "# Creating package.json"
+cat > package.json << 'EOF'
+{
+  "name": "web-capture",
+  "version": "1.0.0",
+  "description": "Microservice to render web pages as HTML, Markdown, or PNG",
+  "main": "index.js",
+  "scripts": {
+    "start": "node index.js",
+    "dev": "nodemon index.js"
+  },
+  "dependencies": {
+    "express": "^4.18.2",
+    "turndown": "^7.1.1",
+    "capture-website": "^4.0.0"
+  },
+  "type": "module",
+  "keywords": ["render", "markdown", "screenshot", "microservice"],
+  "license": "MIT"
+}
+EOF
+
+# Install dependencies
+if command -v yarn >/dev/null 2>&1; then
+  echo "Installing dependencies with yarn..."
+  yarn install
+else
+  echo "Installing dependencies with npm..."
+  npm install
+fi
+
+echo "# Writing index.js"
+cat > index.js << 'EOF'
+import express from 'express';
+import fetch from 'node:fetch';               // Node 18+ has native fetch
+import TurndownService from 'turndown';
+import captureWebsite from 'capture-website';
+
+const app = express();
+const port = process.env.PORT || 3000;
+
+// Route: Fetch raw HTML
+app.get('/html', async (req, res) => {
+  const url = req.query.url;
+  if (!url) return res.status(400).send('Missing `url` parameter');
+  try {
+    const response = await fetch(url);
+    const html = await response.text();
+    res.type('text/html').send(html);
+  } catch (err) {
+    console.error(err);
+    res.status(500).send('Error fetching HTML');
+  }
+});
+
+// Route: Convert HTML to Markdown
+app.get('/markdown', async (req, res) => {
+  const url = req.query.url;
+  if (!url) return res.status(400).send('Missing `url` parameter');
+  try {
+    const response = await fetch(url);
+    const html = await response.text();
+    const turndown = new TurndownService();
+    const markdown = turndown.turndown(html);
+    res.type('text/markdown').send(markdown);
+  } catch (err) {
+    console.error(err);
+    res.status(500).send('Error converting to Markdown');
+  }
+});
+
+// Route: Capture screenshot as PNG
+app.get('/image', async (req, res) => {
+  const url = req.query.url;
+  if (!url) return res.status(400).send('Missing `url` parameter');
+  try {
+    const buffer = await captureWebsite.buffer(url, { fullPage: true });
+    res.type('image/png').send(buffer);
+  } catch (err) {
+    console.error(err);
+    res.status(500).send('Error capturing screenshot');
+  }
+});
+
+app.listen(port, () => {
+  console.log(`Renderer service listening on http://localhost:${port}`);
+});
+EOF
+
+echo "# Writing Dockerfile"
+cat > Dockerfile << 'EOF'
+FROM node:20-slim
+
+WORKDIR /usr/src/app
+COPY package.json yarn.lock* ./
+
+# Install production dependencies and Chrome deps
+RUN apt-get update && apt-get install -y \
+    gconf-service libasound2 libatk1.0-0 libc6 libcairo2 \
+    libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 \
+    libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libstdc++6 \
+    libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 \
+    libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
+    ca-certificates fonts-liberation lsb-release xdg-utils wget \
+    --no-install-recommends && rm -rf /var/lib/apt/lists/* \
+ && npm install --production
+
+COPY . .
+ENV NODE_ENV=production
+EXPOSE 3000
+CMD ["node", "index.js"]
+EOF
+
+echo "# Creating .gitignore"
+cat > .gitignore << 'EOF'
+node_modules
+.env
+EOF
+
+echo "# Creating README.md"
+cat > README.md << 'EOF'
+# web-capture
+
+A microservice to fetch URLs and render them as:
+
+- **HTML**: GET /html?url=<URL>
+- **Markdown**: GET /markdown?url=<URL>
+- **PNG screenshot**: GET /image?url=<URL>
+
+## Usage
+
+```bash
+npm start
+# or
+yarn start
+curl http://localhost:3000/html?url=https://example.com
+```
+
+## Docker
+
+```bash
+docker build -t web-capture .
+docker run -p 3000:3000 web-capture
+```
+EOF
+
+echo "
+✔ Project 'web-capture' scaffolded successfully in $(pwd)
+Next steps:
+  cd "$(pwd)"
+  npm start
+"

package/babel.config.cjs

@@ -0,0 +1,13 @@
+module.exports = {
+  presets: [
+    [
+      '@babel/preset-env',
+      {
+        targets: {
+          node: 'current',
+        },
+        modules: false,
+      },
+    ],
+  ],
+};