pagepull 0.1.0__tar.gz

pagepull-0.1.0/PKG-INFO

@@ -0,0 +1,205 @@
+Metadata-Version: 2.4
+Name: pagepull
+Version: 0.1.0
+Summary: Extract and transform HTML page content with composable CLI tools
+License: MIT
+Author: Neil Johnson
+Author-email: neil@cadent.com
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: beautifulsoup4 (>=4.12,<5.0)
+Requires-Dist: requests (>=2.31,<3.0)
+Description-Content-Type: text/markdown
+
+# pagepull
+
+Extract structured data from HTML pages via the command line.
+
+pagepull wraps [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) behind a simple CLI, turning common DOM extraction tasks into one-liners. Think of it as `jq` for HTML.
+
+## Install
+
+```bash
+pip install pagepull
+```
+
+Or with pipx for isolated install:
+
+```bash
+pipx install pagepull
+```
+
+## Quick Start
+
+```bash
+# Extract the content div from a WordPress page
+pagepull div entry-content https://example.com/about
+
+# Same thing, as markdown
+pagepull div entry-content --markdown https://example.com/about
+
+# List images and check for missing alt text
+pagepull images --alt https://example.com/about
+
+# Pull meta tags
+pagepull meta --title --description https://example.com/about
+
+# Use any CSS selector
+pagepull select "nav.primary a" page.html
+```
+
+## Input
+
+pagepull accepts three input types:
+
+```bash
+# Local file
+pagepull div content page.html
+
+# URL (fetched automatically)
+pagepull div content https://example.com/page
+
+# stdin
+curl -s https://example.com | pagepull div content
+```
+
+## Commands
+
+### `div` — Extract a div by class or id
+
+```bash
+pagepull div entry-content page.html
+pagepull div sidebar --by id page.html
+pagepull div entry-content --strip script,style --markdown page.html
+```
+
+### `images` — List images with metadata
+
+```bash
+pagepull images page.html
+pagepull images --alt --dimensions page.html
+pagepull images --json page.html
+```
+
+Flags `--alt` to show alt text (missing alt flagged as `[MISSING]`) and `--dimensions` for width/height.
+
+### `meta` — Extract meta tags
+
+```bash
+pagepull meta page.html                         # all meta tags
+pagepull meta --title --description page.html   # specific tags
+pagepull meta --og page.html                    # Open Graph tags
+```
+
+### `links` — List all links
+
+```bash
+pagepull links page.html
+pagepull links --external-only page.html
+pagepull links --csv page.html
+```
+
+### `headings` — Heading hierarchy
+
+```bash
+pagepull headings page.html
+```
+
+```
+h1: Welcome to Our Site
+  h2: About Us
+  h2: Services
+    h3: Web Design
+```
+
+### `text` — Visible text only
+
+```bash
+pagepull text page.html
+pagepull text --selector "div.entry-content" page.html
+```
+
+### `select` — Raw CSS selector
+
+```bash
+pagepull select "nav a" page.html
+pagepull select "img[alt='']" --json page.html
+pagepull select "h2 + p" --text page.html
+```
+
+### `strip` — Remove elements
+
+```bash
+pagepull strip script noscript style page.html
+```
+
+### `table` — Extract HTML tables
+
+```bash
+pagepull table --csv page.html
+pagepull table --index 0 --json page.html
+```
+
+## Global Flags
+
+| Flag | Description |
+|------|-------------|
+| `--selector <css>` | Scope any command to a CSS selector first |
+| `--json` | Structured JSON output |
+| `--csv` | CSV output (where applicable) |
+| `--markdown` | Convert HTML to markdown |
+| `--quiet` | Suppress headers and labels |
+
+## Scoping with `--selector`
+
+Any command can be scoped to a portion of the page:
+
+```bash
+# Images only within the article
+pagepull images --alt --selector "article" page.html
+
+# Links only in the footer
+pagepull links --selector "footer" page.html
+
+# Text from a specific section
+pagepull text --selector "div.entry-content" page.html
+```
+
+## Pairing with sitewalker
+
+pagepull handles one page. [sitewalker](https://github.com/cadentdev/sitewalker) crawls sites. Together they cover site-wide extraction:
+
+```bash
+# Audit alt text across an entire site
+sitewalker -p https://example.com | xargs -I{} pagepull images --alt --json {}
+
+# Extract every page title
+sitewalker -p https://example.com | xargs -I{} pagepull meta --title {}
+
+# Pull article content as markdown
+sitewalker -p https://example.com | xargs -I{} pagepull div content --markdown {}
+```
+
+## Development
+
+```bash
+git clone git@github.com:cadentdev/pagepull.git
+cd pagepull
+poetry install
+poetry run pytest
+```
+
+## Requirements
+
+- Python 3.11+
+- Dependencies: beautifulsoup4, requests, markdownify
+
+## License
+
+MIT
+

pagepull-0.1.0/README.md

@@ -0,0 +1,186 @@
+# pagepull
+
+Extract structured data from HTML pages via the command line.
+
+pagepull wraps [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) behind a simple CLI, turning common DOM extraction tasks into one-liners. Think of it as `jq` for HTML.
+
+## Install
+
+```bash
+pip install pagepull
+```
+
+Or with pipx for isolated install:
+
+```bash
+pipx install pagepull
+```
+
+## Quick Start
+
+```bash
+# Extract the content div from a WordPress page
+pagepull div entry-content https://example.com/about
+
+# Same thing, as markdown
+pagepull div entry-content --markdown https://example.com/about
+
+# List images and check for missing alt text
+pagepull images --alt https://example.com/about
+
+# Pull meta tags
+pagepull meta --title --description https://example.com/about
+
+# Use any CSS selector
+pagepull select "nav.primary a" page.html
+```
+
+## Input
+
+pagepull accepts three input types:
+
+```bash
+# Local file
+pagepull div content page.html
+
+# URL (fetched automatically)
+pagepull div content https://example.com/page
+
+# stdin
+curl -s https://example.com | pagepull div content
+```
+
+## Commands
+
+### `div` — Extract a div by class or id
+
+```bash
+pagepull div entry-content page.html
+pagepull div sidebar --by id page.html
+pagepull div entry-content --strip script,style --markdown page.html
+```
+
+### `images` — List images with metadata
+
+```bash
+pagepull images page.html
+pagepull images --alt --dimensions page.html
+pagepull images --json page.html
+```
+
+Flags `--alt` to show alt text (missing alt flagged as `[MISSING]`) and `--dimensions` for width/height.
+
+### `meta` — Extract meta tags
+
+```bash
+pagepull meta page.html                         # all meta tags
+pagepull meta --title --description page.html   # specific tags
+pagepull meta --og page.html                    # Open Graph tags
+```
+
+### `links` — List all links
+
+```bash
+pagepull links page.html
+pagepull links --external-only page.html
+pagepull links --csv page.html
+```
+
+### `headings` — Heading hierarchy
+
+```bash
+pagepull headings page.html
+```
+
+```
+h1: Welcome to Our Site
+  h2: About Us
+  h2: Services
+    h3: Web Design
+```
+
+### `text` — Visible text only
+
+```bash
+pagepull text page.html
+pagepull text --selector "div.entry-content" page.html
+```
+
+### `select` — Raw CSS selector
+
+```bash
+pagepull select "nav a" page.html
+pagepull select "img[alt='']" --json page.html
+pagepull select "h2 + p" --text page.html
+```
+
+### `strip` — Remove elements
+
+```bash
+pagepull strip script noscript style page.html
+```
+
+### `table` — Extract HTML tables
+
+```bash
+pagepull table --csv page.html
+pagepull table --index 0 --json page.html
+```
+
+## Global Flags
+
+| Flag | Description |
+|------|-------------|
+| `--selector <css>` | Scope any command to a CSS selector first |
+| `--json` | Structured JSON output |
+| `--csv` | CSV output (where applicable) |
+| `--markdown` | Convert HTML to markdown |
+| `--quiet` | Suppress headers and labels |
+
+## Scoping with `--selector`
+
+Any command can be scoped to a portion of the page:
+
+```bash
+# Images only within the article
+pagepull images --alt --selector "article" page.html
+
+# Links only in the footer
+pagepull links --selector "footer" page.html
+
+# Text from a specific section
+pagepull text --selector "div.entry-content" page.html
+```
+
+## Pairing with sitewalker
+
+pagepull handles one page. [sitewalker](https://github.com/cadentdev/sitewalker) crawls sites. Together they cover site-wide extraction:
+
+```bash
+# Audit alt text across an entire site
+sitewalker -p https://example.com | xargs -I{} pagepull images --alt --json {}
+
+# Extract every page title
+sitewalker -p https://example.com | xargs -I{} pagepull meta --title {}
+
+# Pull article content as markdown
+sitewalker -p https://example.com | xargs -I{} pagepull div content --markdown {}
+```
+
+## Development
+
+```bash
+git clone git@github.com:cadentdev/pagepull.git
+cd pagepull
+poetry install
+poetry run pytest
+```
+
+## Requirements
+
+- Python 3.11+
+- Dependencies: beautifulsoup4, requests, markdownify
+
+## License
+
+MIT

pagepull-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[tool.poetry]
+name = "pagepull"
+version = "0.1.0"
+description = "Extract and transform HTML page content with composable CLI tools"
+authors = ["Neil Johnson <neil@cadent.com>"]
+license = "MIT"
+readme = "README.md"
+packages = [{include = "pagepull", from = "src"}]
+
+[tool.poetry.dependencies]
+python = "^3.11"
+beautifulsoup4 = "^4.12"
+requests = "^2.31"
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^8.0"
+pytest-cov = "^5.0"
+
+[tool.poetry.scripts]
+pagepull = "pagepull.cli:main"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.coverage.run]
+source = ["pagepull"]

pagepull-0.1.0/src/pagepull/__init__.py

@@ -0,0 +1,3 @@
+"""pagepull — Extract structured data from HTML pages via the CLI."""
+
+__version__ = "0.1.0"

pagepull-0.1.0/src/pagepull/cli.py

@@ -0,0 +1,99 @@
+"""CLI entry point for pagepull."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+from bs4 import BeautifulSoup
+
+from pagepull import __version__
+from pagepull.commands import div, select, strip, text
+from pagepull.output import format_output
+from pagepull.source import load_source
+
+
+def build_parser() -> argparse.ArgumentParser:
+    """Build the argument parser with subcommands."""
+    parser = argparse.ArgumentParser(
+        prog="pagepull",
+        description="Extract structured data from HTML pages.",
+    )
+    parser.add_argument("--version", action="version", version=f"pagepull {__version__}")
+    parser.add_argument("--json", dest="as_json", action="store_true", help="output as JSON")
+    parser.add_argument("--quiet", "-q", action="store_true", help="suppress headers and labels")
+    parser.add_argument("--selector", "-s", metavar="CSS", help="scope to CSS selector before command")
+
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    # div
+    p_div = sub.add_parser("div", help="extract a div by class or id")
+    p_div.add_argument("name", help="class or id name to match")
+    p_div.add_argument("source", nargs="?", default=None, help="file, URL, or omit for stdin")
+    p_div.add_argument("--by", choices=["class", "id"], default="class", help="match by class or id")
+
+    # select
+    p_sel = sub.add_parser("select", help="select elements with CSS selector")
+    p_sel.add_argument("css", help="CSS selector")
+    p_sel.add_argument("source", nargs="?", default=None, help="file, URL, or omit for stdin")
+    p_sel.add_argument("--text", dest="text_only", action="store_true", help="text content only")
+    p_sel.add_argument("--attr", metavar="NAME", help="extract specific attribute")
+
+    # strip
+    p_strip = sub.add_parser("strip", help="remove elements from HTML")
+    p_strip.add_argument("elements", nargs="+", help="tag names to remove")
+    p_strip.add_argument("source", nargs="?", default=None, help="file, URL, or omit for stdin")
+
+    # text
+    p_text = sub.add_parser("text", help="extract visible text")
+    p_text.add_argument("source", nargs="?", default=None, help="file, URL, or omit for stdin")
+
+    return parser
+
+
+def apply_selector(html: str, selector: str) -> str:
+    """Scope HTML to elements matching a CSS selector."""
+    soup = BeautifulSoup(html, "html.parser")
+    matches = soup.select(selector)
+    if not matches:
+        return ""
+    return "\n".join(str(m) for m in matches)
+
+
+def main(argv: list[str] | None = None) -> None:
+    """Main CLI entry point."""
+    parser = build_parser()
+    args = parser.parse_args(argv)
+
+    html = load_source(args.source if hasattr(args, "source") else None)
+
+    if args.selector:
+        html = apply_selector(html, args.selector)
+        if not html:
+            print("No elements matched --selector", file=sys.stderr)
+            sys.exit(1)
+
+    if args.command == "div":
+        result = div.run(html, args.name, by=args.by)
+        if result is None:
+            print(f"Error: div not found: {args.name}", file=sys.stderr)
+            sys.exit(1)
+        output = format_output(result, as_json=args.as_json, quiet=args.quiet, label=f"div.{args.name}")
+
+    elif args.command == "select":
+        result = select.run(html, args.css, text_only=args.text_only, attr=args.attr)
+        output = format_output(result, as_json=args.as_json, quiet=args.quiet, label=f"select: {args.css}")
+
+    elif args.command == "strip":
+        result = strip.run(html, args.elements)
+        output = format_output(result, as_json=args.as_json, quiet=args.quiet)
+
+    elif args.command == "text":
+        result = text.run(html)
+        output = format_output(result, as_json=args.as_json, quiet=args.quiet)
+
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+    print(output)

pagepull-0.1.0/src/pagepull/commands/div.py

@@ -0,0 +1,29 @@
+"""Extract a div by class or id."""
+
+from __future__ import annotations
+
+from bs4 import BeautifulSoup
+
+
+def run(html: str, name: str, by: str = "class") -> str | None:
+    """Extract inner HTML of a div matched by class or id.
+
+    Args:
+        html: HTML content to search.
+        name: Class name or id to match.
+        by: Match method — "class" or "id".
+
+    Returns:
+        Inner HTML of matched div, or None if not found.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+
+    if by == "id":
+        tag = soup.find("div", id=name)
+    else:
+        tag = soup.find("div", class_=name)
+
+    if tag is None:
+        return None
+
+    return tag.decode_contents()

pagepull-0.1.0/src/pagepull/commands/select.py

@@ -0,0 +1,34 @@
+"""Select elements using CSS selectors."""
+
+from __future__ import annotations
+
+from bs4 import BeautifulSoup
+
+
+def run(
+    html: str,
+    selector: str,
+    text_only: bool = False,
+    attr: str | None = None,
+) -> list[str]:
+    """Select elements matching a CSS selector.
+
+    Args:
+        html: HTML content to search.
+        selector: CSS selector string.
+        text_only: If True, return text content instead of HTML.
+        attr: If set, return this attribute's value from each match.
+
+    Returns:
+        List of matched element strings, text content, or attribute values.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+    matches = soup.select(selector)
+
+    if attr:
+        return [tag.get(attr, "") for tag in matches]
+
+    if text_only:
+        return [tag.get_text(strip=True) for tag in matches]
+
+    return [str(tag) for tag in matches]

pagepull-0.1.0/src/pagepull/commands/strip.py

@@ -0,0 +1,24 @@
+"""Remove specified HTML elements and output cleaned HTML."""
+
+from __future__ import annotations
+
+from bs4 import BeautifulSoup
+
+
+def run(html: str, elements: list[str]) -> str:
+    """Remove all instances of specified elements from HTML.
+
+    Args:
+        html: HTML content to clean.
+        elements: Tag names to remove (e.g., ["script", "style"]).
+
+    Returns:
+        Cleaned HTML string.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+
+    for tag_name in elements:
+        for tag in soup.find_all(tag_name):
+            tag.decompose()
+
+    return str(soup)

pagepull-0.1.0/src/pagepull/commands/text.py

@@ -0,0 +1,27 @@
+"""Extract visible text from HTML."""
+
+from __future__ import annotations
+
+from bs4 import BeautifulSoup
+
+
+def run(html: str) -> str:
+    """Extract visible text content, stripping all HTML tags.
+
+    Args:
+        html: HTML content to extract text from.
+
+    Returns:
+        Visible text with whitespace normalized.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+
+    # Remove script and style elements
+    for tag in soup(["script", "style"]):
+        tag.decompose()
+
+    text = soup.get_text(separator="\n")
+
+    # Normalize whitespace: strip each line, remove blank lines
+    lines = [line.strip() for line in text.splitlines()]
+    return "\n".join(line for line in lines if line)

pagepull-0.1.0/src/pagepull/output.py

@@ -0,0 +1,41 @@
+"""Output formatting helpers for JSON, quiet, and text modes."""
+
+from __future__ import annotations
+
+import json
+
+
+def format_output(
+    data: str | list[str] | list[dict],
+    *,
+    as_json: bool = False,
+    quiet: bool = False,
+    label: str = "",
+) -> str:
+    """Format command output for display.
+
+    Args:
+        data: Raw output — string, list of strings, or list of dicts.
+        as_json: Output as JSON.
+        quiet: Suppress labels and headers.
+        label: Header label (ignored if quiet).
+
+    Returns:
+        Formatted string ready for printing.
+    """
+    if as_json:
+        if isinstance(data, str):
+            return json.dumps({"result": data})
+        return json.dumps(data, indent=2)
+
+    if isinstance(data, list):
+        lines = []
+        if label and not quiet:
+            lines.append(f"{label} ({len(data)} found)")
+        for item in data:
+            lines.append(str(item))
+        return "\n".join(lines)
+
+    if label and not quiet:
+        return f"{label}\n{data}"
+    return data

pagepull-0.1.0/src/pagepull/source.py

@@ -0,0 +1,43 @@
+"""Load HTML from file, URL, or stdin."""
+
+from __future__ import annotations
+
+import sys
+
+import requests
+
+
+def load_source(source: str | None) -> str:
+    """Load HTML content from a file path, URL, or stdin.
+
+    Args:
+        source: File path, URL (http/https), or None for stdin.
+
+    Returns:
+        HTML content as string.
+
+    Raises:
+        FileNotFoundError: If file path doesn't exist.
+        SystemExit: If URL fetch fails or stdin is a TTY with no data.
+    """
+    if source is None:
+        if sys.stdin.isatty():
+            print("Error: no input source. Provide a file, URL, or pipe HTML to stdin.", file=sys.stderr)
+            sys.exit(1)
+        return sys.stdin.read()
+
+    if source.startswith(("http://", "https://")):
+        try:
+            resp = requests.get(source, timeout=10)
+            resp.raise_for_status()
+        except requests.RequestException as e:
+            print(f"Error fetching URL: {e}", file=sys.stderr)
+            sys.exit(1)
+        return resp.text
+
+    try:
+        with open(source, encoding="utf-8") as f:
+            return f.read()
+    except FileNotFoundError:
+        print(f"Error: file not found: {source}", file=sys.stderr)
+        sys.exit(1)