webpage-parser 0.1.0__tar.gz → 0.2.0__tar.gz

webpage_parser-0.2.0/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: webpage-parser
+Version: 0.2.0
+Summary: Extract structured fields (title, body, publication time, authors, language and more) from raw HTML pages
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: lxml==6.1.2
+Requires-Dist: trafilatura==2.2.0
+Requires-Dist: dateparser==1.4.2
+
+# webpage-parser
+
+Extract structured fields from a raw HTML page. Given one crawled web record,
+`process_row` returns a fixed set of **35 fields** covering document identity,
+URLs, the main article (title, body, headings, description, authors),
+timestamps, language and content type.
+
+## Install
+
+```bash
+pip install webpage-parser
+```
+
+Python 3.10+ is required (`trafilatura` and `dateparser` need it).
+
+## Python API
+
+```python
+from webpage_parser import process_row
+
+row = {
+    "html": "<html>…</html>",       # the main input
+    "url": "https://example.com/a",  # everything except html is optional
+}
+result = process_row(row)
+print(result["title"], result["published_at"], result["language"])
+```
+
+`process_row(row)` reads the following keys from the input dict. Only `html`
+really matters; without usable HTML most output fields stay empty.
+
+| key | meaning |
+| --- | --- |
+| `html` | raw HTML, the main input |
+| `url`, `durl`, `redirect_url` | candidate page URLs; first non-empty of `redirect_url` > `durl` > `url` wins |
+| `purl` | passed through unchanged |
+| `domain` | registrable-domain hint |
+| `host` | full host-name hint |
+| `headers` | HTTP response headers, a dict or a JSON string |
+| `crawled_time`, `first_found_time`, `last_found_time` | epoch seconds (or milliseconds) |
+| `anchor` | inbound anchor text |
+| `in_domain_links`, `out_domain_links` | link counts; `(0, 0)` is treated as unknown |
+| `raw_id`, `raw_id_version` | opaque document id / version |
+| `extra` | optional dict; recognised sub-keys `time_zone` (offset in hours), `anchor`, `parser_name` |
+| `img_ref` | optional dict of image references |
+
+## Output
+
+`process_row` returns a dict with exactly these 35 keys:
+
+`document_id`, `document_version`, `url`, `url_hash`, `canonical_url`,
+`canonical_url_hash`, `domain`, `registrable_domain`, `purl`, `site_name`,
+`favicon_url`, `host_logo_url`, `image_url`, `title`, `headings`, `body`,
+`description`, `authors`, `published_at`, `published_at_confidence`,
+`published_at_source`, `updated_at`, `language`, `language_confidence`,
+`country_codes`, `content_type`, `mime_type`, `content_return_policy`,
+`parser_version`, `crawled_at`, `first_seen_at`, `last_seen_at`,
+`anchor_text`, `in_domain_links`, `out_domain_links`.
+
+Notes on selected fields:
+
+- `title`, `body`, `description` are strings; `headings` and `authors` are lists.
+- `domain` is the full host name; `registrable_domain` is the eTLD+1.
+- `favicon_url`, `host_logo_url`, `image_url`, `country_codes` and
+  `content_return_policy` are placeholders that are currently always empty.
+- `parser_version` is a constant string.
+
+`empty_result(row)` is also exported and returns the same 35-key dict with the
+HTML-derived fields left empty. `process_row` degrades to it whenever a page
+cannot be parsed, so callers always receive the complete key set.
+
+## Time zone
+
+- `published_at` and `updated_at` are wall-clock strings `YYYY-MM-DD HH:MM:SS`
+  normalised to **UTC+8** (the default assumed site time zone). A value without
+  an explicit offset is interpreted using `extra["time_zone"]`, an offset in
+  hours, falling back to UTC+8 when absent.
+- `crawled_at`, `first_seen_at` and `last_seen_at` are RFC 3339 timestamps in
+  **UTC** with a trailing `Z`, derived from the epoch inputs.
+
+## Command line
+
+```bash
+webpage-parser --input rows.jsonl --output parsed.jsonl
+```
+
+Each input line is a JSON object shaped like the `process_row` argument; each
+output line is the 35-field JSON result.

webpage_parser-0.2.0/README_PYPI.md

@@ -0,0 +1,88 @@
+# webpage-parser
+
+Extract structured fields from a raw HTML page. Given one crawled web record,
+`process_row` returns a fixed set of **35 fields** covering document identity,
+URLs, the main article (title, body, headings, description, authors),
+timestamps, language and content type.
+
+## Install
+
+```bash
+pip install webpage-parser
+```
+
+Python 3.10+ is required (`trafilatura` and `dateparser` need it).
+
+## Python API
+
+```python
+from webpage_parser import process_row
+
+row = {
+    "html": "<html>…</html>",       # the main input
+    "url": "https://example.com/a",  # everything except html is optional
+}
+result = process_row(row)
+print(result["title"], result["published_at"], result["language"])
+```
+
+`process_row(row)` reads the following keys from the input dict. Only `html`
+really matters; without usable HTML most output fields stay empty.
+
+| key | meaning |
+| --- | --- |
+| `html` | raw HTML, the main input |
+| `url`, `durl`, `redirect_url` | candidate page URLs; first non-empty of `redirect_url` > `durl` > `url` wins |
+| `purl` | passed through unchanged |
+| `domain` | registrable-domain hint |
+| `host` | full host-name hint |
+| `headers` | HTTP response headers, a dict or a JSON string |
+| `crawled_time`, `first_found_time`, `last_found_time` | epoch seconds (or milliseconds) |
+| `anchor` | inbound anchor text |
+| `in_domain_links`, `out_domain_links` | link counts; `(0, 0)` is treated as unknown |
+| `raw_id`, `raw_id_version` | opaque document id / version |
+| `extra` | optional dict; recognised sub-keys `time_zone` (offset in hours), `anchor`, `parser_name` |
+| `img_ref` | optional dict of image references |
+
+## Output
+
+`process_row` returns a dict with exactly these 35 keys:
+
+`document_id`, `document_version`, `url`, `url_hash`, `canonical_url`,
+`canonical_url_hash`, `domain`, `registrable_domain`, `purl`, `site_name`,
+`favicon_url`, `host_logo_url`, `image_url`, `title`, `headings`, `body`,
+`description`, `authors`, `published_at`, `published_at_confidence`,
+`published_at_source`, `updated_at`, `language`, `language_confidence`,
+`country_codes`, `content_type`, `mime_type`, `content_return_policy`,
+`parser_version`, `crawled_at`, `first_seen_at`, `last_seen_at`,
+`anchor_text`, `in_domain_links`, `out_domain_links`.
+
+Notes on selected fields:
+
+- `title`, `body`, `description` are strings; `headings` and `authors` are lists.
+- `domain` is the full host name; `registrable_domain` is the eTLD+1.
+- `favicon_url`, `host_logo_url`, `image_url`, `country_codes` and
+  `content_return_policy` are placeholders that are currently always empty.
+- `parser_version` is a constant string.
+
+`empty_result(row)` is also exported and returns the same 35-key dict with the
+HTML-derived fields left empty. `process_row` degrades to it whenever a page
+cannot be parsed, so callers always receive the complete key set.
+
+## Time zone
+
+- `published_at` and `updated_at` are wall-clock strings `YYYY-MM-DD HH:MM:SS`
+  normalised to **UTC+8** (the default assumed site time zone). A value without
+  an explicit offset is interpreted using `extra["time_zone"]`, an offset in
+  hours, falling back to UTC+8 when absent.
+- `crawled_at`, `first_seen_at` and `last_seen_at` are RFC 3339 timestamps in
+  **UTC** with a trailing `Z`, derived from the epoch inputs.
+
+## Command line
+
+```bash
+webpage-parser --input rows.jsonl --output parsed.jsonl
+```
+
+Each input line is a JSON object shaped like the `process_row` argument; each
+output line is the 35-field JSON result.

{webpage_parser-0.1.0 → webpage_parser-0.2.0}/__init__.py

@@ -1,10 +1,10 @@
-"""网页解析算子：从爬取到的 HTML 抽取 title/content/pub_time/author。
+"""网页解析算子：从爬取到的 HTML 原始行抽取结构化字段。
 
 包内模块沿用「相对导入优先、扁平导入兜底」的双模式写法，因此既可以按包导入
 （`from webpage_parser import process_row`），也可以把目录直接摊平当脚本跑
 （`python3 extract.py`）。推荐按包导入。
 """
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 from .pipeline import empty_result, process_row
 

webpage_parser-0.2.0/extract.py

@@ -0,0 +1,42 @@
+"""CLI：逐行将上游网页 JSONL 转换为固定 35 字段结果。"""
+import argparse
+import json
+import sys
+
+try:
+    from .pipeline import process_row, empty_result
+except ImportError:
+    sys.path.insert(0, __file__.rsplit("/", 1)[0])
+    from pipeline import process_row, empty_result
+
+
+def _safe_row(line):
+    try:
+        obj = json.loads(line)
+        return obj if isinstance(obj, dict) else None
+    except Exception:
+        return None
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input", required=True)
+    parser.add_argument("--output", required=True)
+    args = parser.parse_args()
+
+    with open(args.input, "r", encoding="utf-8", errors="replace") as fin, \
+            open(args.output, "w", encoding="utf-8") as fout:
+        for line in fin:
+            if not line.strip():
+                continue
+            row = _safe_row(line)
+            try:
+                result = process_row(row) if row is not None else empty_result()
+            except Exception:
+                result = empty_result(row)
+            fout.write(json.dumps(result, ensure_ascii=False) + "\n")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

{webpage_parser-0.1.0 → webpage_parser-0.2.0}/htmlprep.py

@@ -172,6 +172,7 @@ class PageSignals:
         self.visible_text_len = 0
         self.title_tag = ""
         self.h1_texts = []
+        self.canonical_urls = []
         self.forum = False
         self.ok = False
 
@@ -672,7 +673,13 @@ def _collect_avatar_byline(tree, signals):
 
 
 def _collect_meta(tree, signals):
-    for el in tree.iter("meta"):
+    for el in tree.iter("meta", "link"):
+        if el.tag == "link":
+            rel = (el.get("rel") or "").lower().split()
+            href = clean_text(el.get("href") or "")
+            if "canonical" in rel and href:
+                signals.canonical_urls.append(href)
+            continue
         key = el.get("name") or el.get("property") or el.get("itemprop") or ""
         key = key.strip().lower()  # 保留内部空格：weibo: article:create_at（调研05 坑）
         if not key:

webpage_parser-0.2.0/pipeline.py

@@ -0,0 +1,325 @@
+"""行级编排：一条上游网页记录转换为固定 35 字段输出。"""
+import re
+from datetime import datetime, timedelta
+from urllib.parse import urljoin, urlparse
+
+try:
+    from .util import (clean_text, norm_key, get_host, reg_domain, loop_unescape,
+                       parse_headers, select_url, hash_url, epoch_to_rfc3339,
+                       detect_content_type, detect_language)
+    from .htmlprep import parse_page
+    from .siteconfig import site_config
+    from .pagegate import gate
+    from .title import extract_title
+    from .author import extract_author
+    from .pubtime import extract_pubtime, pubtime_confidence
+    from .content import extract_content
+    from .timetext import parse_absolute
+    from .sitespecific import site_override
+except ImportError:
+    from util import (clean_text, norm_key, get_host, reg_domain, loop_unescape,
+                      parse_headers, select_url, hash_url, epoch_to_rfc3339,
+                      detect_content_type, detect_language)
+    from htmlprep import parse_page
+    from siteconfig import site_config
+    from pagegate import gate
+    from title import extract_title
+    from author import extract_author
+    from pubtime import extract_pubtime, pubtime_confidence
+    from content import extract_content
+    from timetext import parse_absolute
+    from sitespecific import site_override
+
+PARSER_VERSION = "1.0.0"
+OUTPUT_FIELDS = (
+    "document_id", "document_version", "url", "url_hash", "canonical_url",
+    "canonical_url_hash", "domain", "registrable_domain", "purl", "site_name",
+    "favicon_url", "host_logo_url", "image_url", "title", "headings", "body",
+    "description", "authors", "published_at", "published_at_confidence",
+    "published_at_source", "updated_at", "language", "language_confidence",
+    "country_codes", "content_type", "mime_type", "content_return_policy",
+    "parser_version", "crawled_at", "first_seen_at", "last_seen_at",
+    "anchor_text", "in_domain_links", "out_domain_links",
+)
+_SITE_TZ_SKEW = timedelta(minutes=10)
+_HEADING_RE = re.compile(r"^\s{0,3}#{1,6}\s+(.+?)\s*#*\s*$", re.M)
+_SITE_SUFFIX_RE = re.compile(r"^\s*[-–—_|·•:：]+\s*(.+?)\s*$")
+
+
+def _string_or_none(value):
+    if value is None:
+        return None
+    text = str(value).strip()
+    return text or None
+
+
+def _is_zero(value):
+    try:
+        return float(value) == 0
+    except (TypeError, ValueError):
+        return False
+
+
+def empty_result(row=None):
+    row = row if isinstance(row, dict) else {}
+    url = select_url(row)
+    domain = _string_or_none(row.get("host")) or _string_or_none(get_host(url or ""))
+    registrable = _string_or_none(row.get("domain")) or \
+        _string_or_none(reg_domain(domain or ""))
+    in_links = row.get("in_domain_links")
+    out_links = row.get("out_domain_links")
+    if _is_zero(in_links) and _is_zero(out_links):
+        in_links = None
+        out_links = None
+    return {
+        "document_id": _string_or_none(row.get("raw_id")),
+        "document_version": _string_or_none(row.get("raw_id_version")),
+        "url": url,
+        "url_hash": hash_url(url),
+        "canonical_url": None,
+        "canonical_url_hash": None,
+        "domain": domain,
+        "registrable_domain": registrable,
+        "purl": row.get("purl"),
+        "site_name": None,
+        "favicon_url": None,
+        "host_logo_url": None,
+        "image_url": None,
+        "title": None,
+        "headings": [],
+        "body": None,
+        "description": None,
+        "authors": [],
+        "published_at": None,
+        "published_at_confidence": None,
+        "published_at_source": None,
+        "updated_at": None,
+        "language": "und",
+        "language_confidence": 0.0,
+        "country_codes": None,
+        "content_type": None,
+        "mime_type": None,
+        "content_return_policy": None,
+        "parser_version": PARSER_VERSION,
+        "crawled_at": epoch_to_rfc3339(row.get("crawled_time")),
+        "first_seen_at": epoch_to_rfc3339(row.get("first_found_time"), reject_zero=True),
+        "last_seen_at": epoch_to_rfc3339(row.get("last_found_time")),
+        "anchor_text": _string_or_none(row.get("anchor")),
+        "in_domain_links": in_links,
+        "out_domain_links": out_links,
+    }
+
+
+def _visible_text(tree):
+    try:
+        body = tree.find("body")
+        root = body if body is not None else tree
+        for bad in root.iter("script", "style", "noscript", "template"):
+            parent = bad.getparent()
+            if parent is not None:
+                parent.remove(bad)
+        return root.text_content() or ""
+    except Exception:
+        return ""
+
+
+def _parse_crawl_time(row):
+    value = row.get("crawled_time")
+    if value is None:
+        return None
+    if isinstance(value, (int, float)) and not isinstance(value, bool):
+        try:
+            seconds = float(value)
+            if abs(seconds) >= 100_000_000_000:
+                seconds /= 1000.0
+            return datetime.utcfromtimestamp(seconds) + timedelta(hours=8)
+        except (TypeError, ValueError, OverflowError, OSError):
+            return None
+    text = str(value).strip()
+    parsed = parse_absolute(text)
+    return parsed.dt if parsed else None
+
+
+def _first_meta(signals, *keys):
+    for key in keys:
+        for value in signals.meta.get(key, []):
+            text = clean_text(value)
+            if text:
+                return text
+    return None
+
+
+def _canonical_url(signals, page_url):
+    base_url = page_url or ""
+    try:
+        bases = signals.tree.xpath("//base[@href][1]")
+        if bases:
+            candidate_base = urljoin(base_url, clean_text(bases[0].get("href") or ""))
+            parsed_base = urlparse(candidate_base)
+            if parsed_base.scheme.lower() in ("http", "https") and parsed_base.hostname:
+                base_url = candidate_base
+    except Exception:
+        pass
+    candidates = list(signals.canonical_urls)
+    candidates.extend(signals.meta.get("og:url", []))
+    for candidate in candidates:
+        try:
+            resolved = urljoin(base_url, clean_text(candidate))
+            parsed = urlparse(resolved)
+            if parsed.scheme.lower() in ("http", "https") and parsed.hostname:
+                return resolved
+        except Exception:
+            continue
+    return None
+
+
+def _site_name(signals, title, suffix_stripped):
+    explicit = _first_meta(signals, "og:site_name")
+    if explicit:
+        return explicit
+    if not title or not suffix_stripped:
+        return None
+    raw_titles = list(signals.meta.get("og:title", []))
+    if signals.title_tag:
+        raw_titles.append(signals.title_tag)
+    for raw in raw_titles:
+        raw = clean_text(raw)
+        if not raw.startswith(title):
+            continue
+        match = _SITE_SUFFIX_RE.match(raw[len(title):])
+        if match:
+            value = clean_text(match.group(1))
+            if 1 < len(value) <= 80:
+                return value
+    return None
+
+
+def _authors(value):
+    if not value:
+        return []
+    result = []
+    for item in str(value).split("、"):
+        item = clean_text(item)
+        if item and item not in result:
+            result.append(item)
+    return result
+
+
+def _headings(markdown):
+    result = []
+    seen = set()
+    for match in _HEADING_RE.finditer(markdown or ""):
+        value = clean_text(match.group(1))
+        if value and value not in seen:
+            seen.add(value)
+            result.append(value)
+    return result
+
+
+def _declared_language(signals):
+    try:
+        roots = signals.tree.xpath("//html[@lang][1]")
+        if roots:
+            return roots[0].get("lang") or ""
+        return signals.tree.get("lang") or ""
+    except Exception:
+        return ""
+
+
+def process_row(row):
+    out = empty_result(row)
+    raw_html = row.get("html")
+    if not isinstance(raw_html, str):
+        raw_html = ""
+    headers = parse_headers(row.get("headers"))
+    content_type, mime_type = detect_content_type(headers, raw_html)
+    out["content_type"] = content_type
+    out["mime_type"] = mime_type
+    if not raw_html.strip():
+        return out
+    if len(raw_html) > 8_000_000:
+        raw_html = raw_html[:8_000_000]
+
+    url = out["url"] or ""
+    host = get_host(url) or out["domain"] or ""
+    extra = row.get("extra") if isinstance(row.get("extra"), dict) else {}
+    if out["anchor_text"] and not extra.get("anchor"):
+        extra = dict(extra)
+        extra["anchor"] = out["anchor_text"]
+    parser_name = extra.get("parser_name")
+    images = row.get("_IMAGES") if isinstance(row.get("_IMAGES"), list) else []
+    img_ref = row.get("img_ref") if isinstance(row.get("img_ref"), dict) else {}
+    crawl_dt = _parse_crawl_time(row)
+    cfg = site_config(host, parser_name)
+    signals = parse_page(raw_html, url=url, host=host,
+                         collect_state=cfg.allow_l1b,
+                         forum=cfg.page_type in ("forum_page", "comment_page"))
+    if not signals.ok:
+        return out
+
+    canonical = _canonical_url(signals, url)
+    out["canonical_url"] = canonical
+    out["canonical_url_hash"] = hash_url(canonical)
+    out["description"] = _first_meta(signals, "description", "og:description")
+    out["site_name"] = _first_meta(signals, "og:site_name")
+
+    page_status, _ = gate(signals, cfg, headers)
+    visible_text = _visible_text(signals.tree)
+    if page_status == "list_page":
+        language, confidence = detect_language(visible_text, _declared_language(signals))
+        out["language"] = language
+        out["language_confidence"] = confidence
+        return out
+    visible_norm = norm_key(visible_text)
+    html_norm = norm_key(loop_unescape(raw_html[:1_500_000]))
+
+    title, _, suffix_stripped = extract_title(signals, cfg, html_norm)
+    author, _, _ = extract_author(
+        signals, cfg, extra, visible_norm, html_norm, title=title)
+
+    anchor = ""
+    if cfg.allow_anchor_author:
+        anchor = clean_text(str(extra.get("anchor") or ""))
+    pubtime = extract_pubtime(signals, cfg, crawl_dt, extra, headers, url=url,
+                              anchor=anchor)
+
+    if page_status == "spa_empty":
+        body = ""
+    else:
+        body, _, _, _ = extract_content(
+            signals, cfg, page_status, images, img_ref, visible_norm,
+            title=title, url=url, pub_time=pubtime["pub_time"])
+
+    override = site_override(host, signals, extra)
+    if override:
+        if override.get("content"):
+            body = override["content"]
+        if override.get("title"):
+            title = override["title"]
+        if override.get("author") and not author:
+            author = override["author"]
+        if override.get("pub_time_raw") and not pubtime["pub_time"]:
+            parsed = parse_absolute(override["pub_time_raw"])
+            if parsed and (not crawl_dt or parsed.dt <= crawl_dt + _SITE_TZ_SKEW):
+                pubtime["pub_time"] = parsed.fmt()
+                pubtime["pub_time_source"] = "site_rule"
+                pubtime["pub_time_precision"] = parsed.precision
+                pubtime["pub_time_tz_offset"] = parsed.tz_offset
+                pubtime["pub_time_raw"] = parsed.raw[:120]
+
+    out["title"] = title or None
+    out["body"] = body or None
+    out["authors"] = _authors(author)
+    out["headings"] = _headings(body)
+    if not out["site_name"]:
+        out["site_name"] = _site_name(signals, title, suffix_stripped)
+    out["published_at"] = pubtime["pub_time"] or None
+    out["published_at_source"] = pubtime["pub_time_source"] \
+        if pubtime["pub_time_source"] != "none" else None
+    out["published_at_confidence"] = pubtime_confidence(pubtime)
+    out["updated_at"] = pubtime["updated_at"]
+    language, confidence = detect_language(body or visible_text or title,
+                                           _declared_language(signals))
+    out["language"] = language
+    out["language_confidence"] = confidence
+    return out