webpage-parser 0.2.0__tar.gz → 0.3.0__tar.gz

webpage_parser-0.3.0/PKG-INFO

@@ -0,0 +1,193 @@
+Metadata-Version: 2.4
+Name: webpage-parser
+Version: 0.3.0
+Summary: Extract structured fields (title, body, publication time, authors, language and more) from raw HTML pages
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: beautifulsoup4==4.15.0
+Requires-Dist: lxml==6.1.2
+Requires-Dist: tldextract==5.3.2
+
+# webpage-parser
+
+Extract structured fields from a raw HTML page. Given one crawled web record,
+`process_row` returns a fixed set of **35 fields** covering document identity,
+URLs, the main article (title, body, headings, description, authors),
+timestamps, language and content type.
+
+## Install
+
+```bash
+pip install webpage-parser
+```
+
+Python 3.10+ is required. (The dependencies themselves only need 3.9 at most;
+the floor is aligned with the cp311 runtime used to build the MaxCompute UDF.)
+
+## Python API
+
+`process_row` is the only entry point: hand it one dict, get back a dict with
+exactly 35 keys.
+
+```python
+import json
+
+from webpage_parser import empty_result, process_row
+
+HTML = """<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>Quantum computing reaches 100 qubits - Example Times</title>
+  <meta name="description" content="Researchers entangled 100 superconducting qubits.">
+  <meta property="og:site_name" content="Example Times">
+  <meta name="author" content="Jane Doe">
+  <meta property="article:published_time" content="2026-09-05T10:30:00+00:00">
+  <link rel="canonical" href="https://tech.example.com/news/quantum-2026">
+</head>
+<body>
+  <article>
+    <h1>Quantum computing reaches 100 qubits</h1>
+    <p>The team said a superconducting chip held a stable entangled state across
+       100 qubits, tripling coherence time compared with the previous generation.</p>
+    <h2>How it works</h2>
+    <p>Tunable couplers suppress crosstalk, and dynamical decoupling extends the
+       coherence time.</p>
+  </article>
+</body>
+</html>
+"""
+
+# 1) Minimal call: html is the only key that really matters
+result = process_row({"html": HTML})
+print(result["title"], result["language"], len(result))
+# Quantum computing reaches 100 qubits en 35
+
+# 2) Full call: every key the parser looks at
+row = {
+    "raw_id": "doc-0001",
+    "raw_id_version": "v1",
+    "url": "https://tech.example.com/news/quantum?utm_source=x",
+    "durl": None,
+    "redirect_url": None,
+    "purl": "https://tech.example.com/",
+    "domain": "example.com",
+    "host": "tech.example.com",
+    "html": HTML,
+    "headers": '{"Content-Type": "text/html; charset=utf-8"}',
+    "crawled_time": 1789000000,
+    "first_found_time": 1788000000,
+    "last_found_time": 1789000000,
+    "anchor": "quantum milestone",
+    "in_domain_links": 12,
+    "out_domain_links": 3,
+}
+out = process_row(row)
+
+# 3) Batch: one JSON object per line in, one 35-field object per line out
+with open("parsed.jsonl", "w") as f:
+    for one in [row]:
+        f.write(json.dumps(process_row(one), ensure_ascii=False) + "\n")
+
+# 4) Fallback: the empty result carries the very same key set
+blank = empty_result(row)
+assert set(blank) == set(out)
+```
+
+What `out` actually holds:
+
+```text
+document_id             = 'doc-0001'
+document_version        = 'v1'
+url                     = 'https://tech.example.com/news/quantum?utm_source=x'
+canonical_url           = 'https://tech.example.com/news/quantum-2026'
+domain                  = 'tech.example.com'
+registrable_domain      = 'example.com'
+site_name               = 'Example Times'
+title                   = 'Quantum computing reaches 100 qubits'
+headings                = ['How it works']
+description             = 'Researchers entangled 100 superconducting qubits.'
+authors                 = ['Jane Doe']
+published_at            = '2026-09-05T10:30:00+00:00'
+published_at_source     = 'meta.article_published_time'
+published_at_confidence = 0.95
+language                = 'en'
+language_confidence     = 0.7
+content_type            = 'html'
+mime_type               = 'text/html'
+crawled_at              = '2026-09-10T00:26:40Z'
+first_seen_at           = '2026-08-29T10:40:00Z'
+last_seen_at            = '2026-09-10T00:26:40Z'
+anchor_text             = 'quantum milestone'
+in_domain_links         = 12
+out_domain_links        = 3
+```
+
+Four things that tend to surprise people:
+
+- `title` drops the trailing site suffix (`- Example Times`); the site name is
+  reported separately as `site_name`.
+- The `h1` was taken as the title, so it is not repeated in `headings`.
+- `published_at` keeps the page's own UTC offset. The page declares
+  `10:30:00+00:00`, so the output stays `2026-09-05T10:30:00+00:00`; see the
+  Time zone section below.
+- Every call returns all 35 keys, parsed or not.
+
+`process_row(row)` reads the following keys from the input dict. Only `html`
+really matters; without usable HTML most output fields stay empty.
+
+| key | meaning |
+| --- | --- |
+| `html` | raw HTML, the main input |
+| `url`, `durl`, `redirect_url` | candidate page URLs; first non-empty of `redirect_url` > `durl` > `url` wins |
+| `purl` | passed through unchanged |
+| `domain` | registrable-domain hint |
+| `host` | full host-name hint |
+| `headers` | HTTP response headers, a dict or a JSON string |
+| `crawled_time`, `first_found_time`, `last_found_time` | epoch seconds (or milliseconds) |
+| `anchor` | inbound anchor text |
+| `in_domain_links`, `out_domain_links` | link counts; `(0, 0)` is treated as unknown |
+| `raw_id`, `raw_id_version` | opaque document id / version |
+
+## Output
+
+`process_row` returns a dict with exactly these 35 keys:
+
+`document_id`, `document_version`, `url`, `url_hash`, `canonical_url`,
+`canonical_url_hash`, `domain`, `registrable_domain`, `purl`, `site_name`,
+`favicon_url`, `host_logo_url`, `image_url`, `title`, `headings`, `body`,
+`description`, `authors`, `published_at`, `published_at_confidence`,
+`published_at_source`, `updated_at`, `language`, `language_confidence`,
+`country_codes`, `content_type`, `mime_type`, `content_return_policy`,
+`parser_version`, `crawled_at`, `first_seen_at`, `last_seen_at`,
+`anchor_text`, `in_domain_links`, `out_domain_links`.
+
+Notes on selected fields:
+
+- `title`, `body`, `description` are strings; `headings` and `authors` are lists.
+- `domain` is the full host name; `registrable_domain` is the eTLD+1.
+- `favicon_url`, `host_logo_url`, `image_url`, `country_codes` and
+  `content_return_policy` are placeholders that are currently always `None`.
+- `parser_version` is a constant string (currently `'1.1.0'`).
+
+`empty_result(row)` is also exported and returns the same 35-key dict with the
+HTML-derived fields left empty. `process_row` degrades to it whenever a page
+cannot be parsed, so callers always receive the complete key set.
+
+## Time zone
+
+- `published_at` and `updated_at` are ISO 8601 strings that **preserve the
+  page's own UTC offset** — `Z` stays `Z`, `+00:00` stays `+00:00`, `+08:00`
+  stays `+08:00`. When the page carries no explicit offset, **UTC+8** is
+  assumed (a date-only value becomes `...T00:00:00+08:00`).
+- `crawled_at`, `first_seen_at` and `last_seen_at` are RFC 3339 timestamps in
+  **UTC** with a trailing `Z`, derived from the epoch inputs.
+
+## Command line
+
+```bash
+webpage-parser --input rows.jsonl --output parsed.jsonl
+```
+
+Each input line is a JSON object shaped like the `process_row` argument; each
+output line is the 35-field JSON result.

webpage_parser-0.3.0/README_PYPI.md

@@ -0,0 +1,183 @@
+# webpage-parser
+
+Extract structured fields from a raw HTML page. Given one crawled web record,
+`process_row` returns a fixed set of **35 fields** covering document identity,
+URLs, the main article (title, body, headings, description, authors),
+timestamps, language and content type.
+
+## Install
+
+```bash
+pip install webpage-parser
+```
+
+Python 3.10+ is required. (The dependencies themselves only need 3.9 at most;
+the floor is aligned with the cp311 runtime used to build the MaxCompute UDF.)
+
+## Python API
+
+`process_row` is the only entry point: hand it one dict, get back a dict with
+exactly 35 keys.
+
+```python
+import json
+
+from webpage_parser import empty_result, process_row
+
+HTML = """<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>Quantum computing reaches 100 qubits - Example Times</title>
+  <meta name="description" content="Researchers entangled 100 superconducting qubits.">
+  <meta property="og:site_name" content="Example Times">
+  <meta name="author" content="Jane Doe">
+  <meta property="article:published_time" content="2026-09-05T10:30:00+00:00">
+  <link rel="canonical" href="https://tech.example.com/news/quantum-2026">
+</head>
+<body>
+  <article>
+    <h1>Quantum computing reaches 100 qubits</h1>
+    <p>The team said a superconducting chip held a stable entangled state across
+       100 qubits, tripling coherence time compared with the previous generation.</p>
+    <h2>How it works</h2>
+    <p>Tunable couplers suppress crosstalk, and dynamical decoupling extends the
+       coherence time.</p>
+  </article>
+</body>
+</html>
+"""
+
+# 1) Minimal call: html is the only key that really matters
+result = process_row({"html": HTML})
+print(result["title"], result["language"], len(result))
+# Quantum computing reaches 100 qubits en 35
+
+# 2) Full call: every key the parser looks at
+row = {
+    "raw_id": "doc-0001",
+    "raw_id_version": "v1",
+    "url": "https://tech.example.com/news/quantum?utm_source=x",
+    "durl": None,
+    "redirect_url": None,
+    "purl": "https://tech.example.com/",
+    "domain": "example.com",
+    "host": "tech.example.com",
+    "html": HTML,
+    "headers": '{"Content-Type": "text/html; charset=utf-8"}',
+    "crawled_time": 1789000000,
+    "first_found_time": 1788000000,
+    "last_found_time": 1789000000,
+    "anchor": "quantum milestone",
+    "in_domain_links": 12,
+    "out_domain_links": 3,
+}
+out = process_row(row)
+
+# 3) Batch: one JSON object per line in, one 35-field object per line out
+with open("parsed.jsonl", "w") as f:
+    for one in [row]:
+        f.write(json.dumps(process_row(one), ensure_ascii=False) + "\n")
+
+# 4) Fallback: the empty result carries the very same key set
+blank = empty_result(row)
+assert set(blank) == set(out)
+```
+
+What `out` actually holds:
+
+```text
+document_id             = 'doc-0001'
+document_version        = 'v1'
+url                     = 'https://tech.example.com/news/quantum?utm_source=x'
+canonical_url           = 'https://tech.example.com/news/quantum-2026'
+domain                  = 'tech.example.com'
+registrable_domain      = 'example.com'
+site_name               = 'Example Times'
+title                   = 'Quantum computing reaches 100 qubits'
+headings                = ['How it works']
+description             = 'Researchers entangled 100 superconducting qubits.'
+authors                 = ['Jane Doe']
+published_at            = '2026-09-05T10:30:00+00:00'
+published_at_source     = 'meta.article_published_time'
+published_at_confidence = 0.95
+language                = 'en'
+language_confidence     = 0.7
+content_type            = 'html'
+mime_type               = 'text/html'
+crawled_at              = '2026-09-10T00:26:40Z'
+first_seen_at           = '2026-08-29T10:40:00Z'
+last_seen_at            = '2026-09-10T00:26:40Z'
+anchor_text             = 'quantum milestone'
+in_domain_links         = 12
+out_domain_links        = 3
+```
+
+Four things that tend to surprise people:
+
+- `title` drops the trailing site suffix (`- Example Times`); the site name is
+  reported separately as `site_name`.
+- The `h1` was taken as the title, so it is not repeated in `headings`.
+- `published_at` keeps the page's own UTC offset. The page declares
+  `10:30:00+00:00`, so the output stays `2026-09-05T10:30:00+00:00`; see the
+  Time zone section below.
+- Every call returns all 35 keys, parsed or not.
+
+`process_row(row)` reads the following keys from the input dict. Only `html`
+really matters; without usable HTML most output fields stay empty.
+
+| key | meaning |
+| --- | --- |
+| `html` | raw HTML, the main input |
+| `url`, `durl`, `redirect_url` | candidate page URLs; first non-empty of `redirect_url` > `durl` > `url` wins |
+| `purl` | passed through unchanged |
+| `domain` | registrable-domain hint |
+| `host` | full host-name hint |
+| `headers` | HTTP response headers, a dict or a JSON string |
+| `crawled_time`, `first_found_time`, `last_found_time` | epoch seconds (or milliseconds) |
+| `anchor` | inbound anchor text |
+| `in_domain_links`, `out_domain_links` | link counts; `(0, 0)` is treated as unknown |
+| `raw_id`, `raw_id_version` | opaque document id / version |
+
+## Output
+
+`process_row` returns a dict with exactly these 35 keys:
+
+`document_id`, `document_version`, `url`, `url_hash`, `canonical_url`,
+`canonical_url_hash`, `domain`, `registrable_domain`, `purl`, `site_name`,
+`favicon_url`, `host_logo_url`, `image_url`, `title`, `headings`, `body`,
+`description`, `authors`, `published_at`, `published_at_confidence`,
+`published_at_source`, `updated_at`, `language`, `language_confidence`,
+`country_codes`, `content_type`, `mime_type`, `content_return_policy`,
+`parser_version`, `crawled_at`, `first_seen_at`, `last_seen_at`,
+`anchor_text`, `in_domain_links`, `out_domain_links`.
+
+Notes on selected fields:
+
+- `title`, `body`, `description` are strings; `headings` and `authors` are lists.
+- `domain` is the full host name; `registrable_domain` is the eTLD+1.
+- `favicon_url`, `host_logo_url`, `image_url`, `country_codes` and
+  `content_return_policy` are placeholders that are currently always `None`.
+- `parser_version` is a constant string (currently `'1.1.0'`).
+
+`empty_result(row)` is also exported and returns the same 35-key dict with the
+HTML-derived fields left empty. `process_row` degrades to it whenever a page
+cannot be parsed, so callers always receive the complete key set.
+
+## Time zone
+
+- `published_at` and `updated_at` are ISO 8601 strings that **preserve the
+  page's own UTC offset** — `Z` stays `Z`, `+00:00` stays `+00:00`, `+08:00`
+  stays `+08:00`. When the page carries no explicit offset, **UTC+8** is
+  assumed (a date-only value becomes `...T00:00:00+08:00`).
+- `crawled_at`, `first_seen_at` and `last_seen_at` are RFC 3339 timestamps in
+  **UTC** with a trailing `Z`, derived from the epoch inputs.
+
+## Command line
+
+```bash
+webpage-parser --input rows.jsonl --output parsed.jsonl
+```
+
+Each input line is a JSON object shaped like the `process_row` argument; each
+output line is the 35-field JSON result.

webpage_parser-0.3.0/__init__.py

@@ -0,0 +1,11 @@
+"""网页解析算子：从爬取到的 HTML 原始行抽取结构化字段。
+
+解析逻辑已全部合并进 extract.py，既可以按包导入（`from webpage_parser import
+process_row`），也可以把 extract.py 当脚本跑（`python3 extract.py`）。推荐按包导入。
+"""
+__version__ = "0.3.0"
+
+# 历史 empty_result(row) 现由 extract.build_base 提供，保留别名以稳定对外 API
+from .extract import build_base as empty_result, process_row
+
+__all__ = ["empty_result", "process_row", "__version__"]