daz-web-extract 0.2.0__tar.gz

daz_web_extract-0.2.0/PKG-INFO

@@ -0,0 +1,186 @@
+Metadata-Version: 2.4
+Name: daz-web-extract
+Version: 0.2.0
+Summary: Async web content extraction library with three-tier fetch strategy
+Author-email: Darren Oakey <darren@oakey.net>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/darrenoakey/daz-web-extract
+Project-URL: Repository, https://github.com/darrenoakey/daz-web-extract
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: httpx>=0.28.0
+Requires-Dist: lxml>=6.0.0
+Requires-Dist: trafilatura>=1.6.0
+Requires-Dist: playwright>=1.40.0
+Requires-Dist: setproctitle>=1.3.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+
+![](banner.jpg)
+
+# daz-web-extract
+
+Async Python library that extracts clean title and body text from any URL. It automatically escalates through multiple fetch strategies to handle everything from simple static pages to JavaScript-rendered content. It never throws exceptions — every call returns a structured result indicating success or failure.
+
+## Installation
+
+Requires Python 3.12+.
+
+```bash
+pip install daz-web-extract
+```
+
+After installing, set up the browser engine for pages that require JavaScript rendering:
+
+```bash
+playwright install chromium
+```
+
+## Usage
+
+### Python API
+
+The library exposes a single async function `extract` and a result type `ExtractionResult`.
+
+```python
+import asyncio
+from daz_web_extract import extract, ExtractionResult
+
+result: ExtractionResult = asyncio.run(extract("https://example.com"))
+
+if result.success:
+    print(result.title)           # Page title
+    print(result.body)            # Clean body text
+    print(result.fetch_method)    # Which strategy succeeded
+    print(result.content_length)  # Length of body in characters
+    print(result.elapsed_ms)      # Total time in milliseconds
+    print(result.status_code)     # HTTP status code (if available)
+else:
+    print(result.error)           # Human-readable error message
+```
+
+#### Limiting fetch strategies
+
+Use the `max_tier` parameter to control how far the library escalates:
+
+```python
+# Only use fast HTTP fetch (no browser, no trafilatura)
+result = await extract("https://example.com", max_tier=1)
+
+# Use HTTP fetch + trafilatura, but skip the browser
+result = await extract("https://example.com", max_tier=2)
+
+# Use all strategies including headless browser (default)
+result = await extract("https://example.com", max_tier=3)
+```
+
+#### Serialization
+
+Results can be converted to dictionaries or JSON:
+
+```python
+result.to_dict()  # Returns a plain dict
+result.to_json()  # Returns a JSON string
+```
+
+#### Using in async code
+
+```python
+import asyncio
+from daz_web_extract import extract
+
+async def main():
+    urls = [
+        "https://example.com",
+        "https://www.iana.org/help/example-domains",
+    ]
+    results = await asyncio.gather(*[extract(url) for url in urls])
+    for r in results:
+        print(f"{r.url}: {r.title} ({r.content_length} chars)")
+
+asyncio.run(main())
+```
+
+### Command Line
+
+Extract content from a URL and print the result:
+
+```bash
+python run_cli.py extract https://example.com
+```
+
+Output:
+
+```
+Title: Example Domain
+Method: httpx
+Length: 217 chars
+Time: 142ms
+
+Example Domain
+This domain is for use in illustrative examples in documents. You may use this domain
+in literature without prior coordination or asking for permission.
+More information...
+```
+
+Get raw JSON output:
+
+```bash
+python run_cli.py extract https://example.com --raw
+```
+
+Output:
+
+```json
+{
+  "success": true,
+  "url": "https://example.com",
+  "title": "Example Domain",
+  "body": "Example Domain\nThis domain is for use in ...",
+  "error": null,
+  "fetch_method": "httpx",
+  "status_code": 200,
+  "content_length": 217,
+  "elapsed_ms": 142
+}
+```
+
+### Using via the `run` script
+
+The project includes a `run` script that automatically activates the virtual environment:
+
+```bash
+# Extract content
+./run extract https://example.com
+./run extract https://example.com --raw
+
+# Run tests
+./run test src/daz_web_extract/result_test.py
+
+# Run linter
+./run lint
+
+# Run full quality checks
+./run check
+```
+
+## Development
+
+Set up a development environment:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+playwright install chromium
+```
+
+Run the tests:
+
+```bash
+pytest -q src/
+```

daz_web_extract-0.2.0/README.md

@@ -0,0 +1,164 @@
+![](banner.jpg)
+
+# daz-web-extract
+
+Async Python library that extracts clean title and body text from any URL. It automatically escalates through multiple fetch strategies to handle everything from simple static pages to JavaScript-rendered content. It never throws exceptions — every call returns a structured result indicating success or failure.
+
+## Installation
+
+Requires Python 3.12+.
+
+```bash
+pip install daz-web-extract
+```
+
+After installing, set up the browser engine for pages that require JavaScript rendering:
+
+```bash
+playwright install chromium
+```
+
+## Usage
+
+### Python API
+
+The library exposes a single async function `extract` and a result type `ExtractionResult`.
+
+```python
+import asyncio
+from daz_web_extract import extract, ExtractionResult
+
+result: ExtractionResult = asyncio.run(extract("https://example.com"))
+
+if result.success:
+    print(result.title)           # Page title
+    print(result.body)            # Clean body text
+    print(result.fetch_method)    # Which strategy succeeded
+    print(result.content_length)  # Length of body in characters
+    print(result.elapsed_ms)      # Total time in milliseconds
+    print(result.status_code)     # HTTP status code (if available)
+else:
+    print(result.error)           # Human-readable error message
+```
+
+#### Limiting fetch strategies
+
+Use the `max_tier` parameter to control how far the library escalates:
+
+```python
+# Only use fast HTTP fetch (no browser, no trafilatura)
+result = await extract("https://example.com", max_tier=1)
+
+# Use HTTP fetch + trafilatura, but skip the browser
+result = await extract("https://example.com", max_tier=2)
+
+# Use all strategies including headless browser (default)
+result = await extract("https://example.com", max_tier=3)
+```
+
+#### Serialization
+
+Results can be converted to dictionaries or JSON:
+
+```python
+result.to_dict()  # Returns a plain dict
+result.to_json()  # Returns a JSON string
+```
+
+#### Using in async code
+
+```python
+import asyncio
+from daz_web_extract import extract
+
+async def main():
+    urls = [
+        "https://example.com",
+        "https://www.iana.org/help/example-domains",
+    ]
+    results = await asyncio.gather(*[extract(url) for url in urls])
+    for r in results:
+        print(f"{r.url}: {r.title} ({r.content_length} chars)")
+
+asyncio.run(main())
+```
+
+### Command Line
+
+Extract content from a URL and print the result:
+
+```bash
+python run_cli.py extract https://example.com
+```
+
+Output:
+
+```
+Title: Example Domain
+Method: httpx
+Length: 217 chars
+Time: 142ms
+
+Example Domain
+This domain is for use in illustrative examples in documents. You may use this domain
+in literature without prior coordination or asking for permission.
+More information...
+```
+
+Get raw JSON output:
+
+```bash
+python run_cli.py extract https://example.com --raw
+```
+
+Output:
+
+```json
+{
+  "success": true,
+  "url": "https://example.com",
+  "title": "Example Domain",
+  "body": "Example Domain\nThis domain is for use in ...",
+  "error": null,
+  "fetch_method": "httpx",
+  "status_code": 200,
+  "content_length": 217,
+  "elapsed_ms": 142
+}
+```
+
+### Using via the `run` script
+
+The project includes a `run` script that automatically activates the virtual environment:
+
+```bash
+# Extract content
+./run extract https://example.com
+./run extract https://example.com --raw
+
+# Run tests
+./run test src/daz_web_extract/result_test.py
+
+# Run linter
+./run lint
+
+# Run full quality checks
+./run check
+```
+
+## Development
+
+Set up a development environment:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+playwright install chromium
+```
+
+Run the tests:
+
+```bash
+pytest -q src/
+```

daz_web_extract-0.2.0/pyproject.toml

@@ -0,0 +1,46 @@
+[build-system]
+requires = ["setuptools>=68.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "daz-web-extract"
+version = "0.2.0"
+description = "Async web content extraction library with three-tier fetch strategy"
+requires-python = ">=3.12"
+license = "MIT"
+authors = [
+    {name = "Darren Oakey", email = "darren@oakey.net"},
+]
+readme = "README.md"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    "httpx>=0.28.0",
+    "lxml>=6.0.0",
+    "trafilatura>=1.6.0",
+    "playwright>=1.40.0",
+    "setproctitle>=1.3.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/darrenoakey/daz-web-extract"
+Repository = "https://github.com/darrenoakey/daz-web-extract"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.4.0",
+    "pytest-asyncio>=0.21.0",
+    "ruff>=0.1.0",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["src"]
+
+[tool.ruff]
+line-length = 120

daz_web_extract-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

daz_web_extract-0.2.0/src/daz_web_extract/__init__.py

@@ -0,0 +1,4 @@
+from daz_web_extract.result import ExtractionResult
+from daz_web_extract.extract import extract
+
+__all__ = ["extract", "ExtractionResult"]

daz_web_extract-0.2.0/src/daz_web_extract/content.py

@@ -0,0 +1,134 @@
+from __future__ import annotations
+
+import re
+
+import lxml.html
+
+
+NOISE_TAGS = {
+    "script", "style", "nav", "footer", "aside", "header", "noscript",
+    "iframe", "form", "svg", "button", "select", "option", "textarea",
+    "input", "label", "fieldset", "legend", "dialog", "menu", "menuitem",
+    "details", "summary",
+}
+NOISE_CLASSES = {
+    "ad", "ads", "advert", "advertisement", "banner", "sponsor", "sponsored",
+    "promo", "promotion", "sidebar", "widget", "social", "share", "sharing",
+    "cookie", "consent", "popup", "modal", "overlay", "newsletter",
+    "subscribe", "signup", "sign-up", "cta", "call-to-action",
+    "related", "recommended", "trending", "popular", "breadcrumb",
+    "pagination", "pager", "toolbar", "tooltip", "dropdown",
+    "comment", "comments", "disqus",
+}
+NOISE_IDS = {
+    "ad", "ads", "sidebar", "cookie-banner", "newsletter",
+    "comments", "disqus_thread", "social-share",
+}
+NOISE_ROLES = {"navigation", "banner", "complementary", "contentinfo", "form", "search", "menu", "menubar"}
+CONTENT_TAGS = {"p", "h1", "h2", "h3", "h4", "h5", "h6", "li", "blockquote", "td", "th", "figcaption", "pre", "dd"}
+MIN_BLOCK_LENGTH = 15
+MIN_BODY_LENGTH = 100
+TITLE_SUFFIX_RE = re.compile(r"\s*[\|\-\u2013\u2014]\s*[^|\-\u2013\u2014]+$")
+
+
+# ##################################################################
+# parse html
+# convert raw html bytes or string into an lxml element tree
+def parse_html(html: str | bytes) -> lxml.html.HtmlElement:
+    if isinstance(html, bytes):
+        html = html.decode("utf-8", errors="replace")
+    return lxml.html.fromstring(html)
+
+
+# ##################################################################
+# extract title
+# pull the best title from html using priority: og:title > <title> > first h1
+def extract_title(tree: lxml.html.HtmlElement) -> str | None:
+    og = tree.xpath('//meta[@property="og:title"]/@content')
+    if og and og[0].strip():
+        return og[0].strip()
+
+    title_els = tree.xpath("//title/text()")
+    if title_els:
+        raw = title_els[0].strip()
+        if raw:
+            return _clean_title_suffix(raw)
+
+    h1_els = tree.xpath("//h1//text()")
+    if h1_els:
+        combined = " ".join(t.strip() for t in h1_els if t.strip())
+        if combined:
+            return combined
+
+    return None
+
+
+# ##################################################################
+# clean title suffix
+# remove common site name suffixes like " | SiteName" or " - SiteName"
+def _clean_title_suffix(title: str) -> str:
+    cleaned = TITLE_SUFFIX_RE.sub("", title)
+    return cleaned if cleaned.strip() else title
+
+
+# ##################################################################
+# extract text content
+# pull clean article text from html by collecting text from content
+# tags after removing all noise elements; link text is preserved but
+# all html formatting is stripped to produce plain text
+def extract_text_content(tree: lxml.html.HtmlElement) -> str | None:
+    _remove_noise(tree)
+    blocks = _collect_blocks(tree)
+    filtered = [b for b in blocks if len(b) >= MIN_BLOCK_LENGTH]
+    body = "\n\n".join(filtered)
+    if len(body) < MIN_BODY_LENGTH:
+        return None
+    return body
+
+
+# ##################################################################
+# remove noise
+# strip script, style, nav, footer, ads, forms, and other noise
+# elements from tree using tags, class names, ids, and aria roles
+def _remove_noise(tree: lxml.html.HtmlElement) -> None:
+    to_remove = []
+    for el in tree.xpath("//*"):
+        if _is_noise_element(el):
+            to_remove.append(el)
+    for el in to_remove:
+        parent = el.getparent()
+        if parent is not None:
+            parent.remove(el)
+
+
+# ##################################################################
+# is noise element
+# check whether an element is noise by tag, class, id, or role
+def _is_noise_element(el: lxml.html.HtmlElement) -> bool:
+    if el.tag in NOISE_TAGS:
+        return True
+    classes = set(el.get("class", "").lower().split())
+    if classes & NOISE_CLASSES:
+        return True
+    el_id = el.get("id", "").lower()
+    if el_id in NOISE_IDS:
+        return True
+    role = el.get("role", "").lower()
+    if role in NOISE_ROLES:
+        return True
+    return False
+
+
+# ##################################################################
+# collect blocks
+# gather text content from paragraph-like elements; link text within
+# paragraphs is preserved since links are part of article content
+def _collect_blocks(tree: lxml.html.HtmlElement) -> list[str]:
+    blocks: list[str] = []
+    for el in tree.xpath("//*"):
+        if el.tag in CONTENT_TAGS:
+            text = el.text_content().strip()
+            text = re.sub(r"\s+", " ", text)
+            if text:
+                blocks.append(text)
+    return blocks