webpage-parser 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (25) hide show
  1. webpage_parser-0.1.0/MANIFEST.in +8 -0
  2. webpage_parser-0.1.0/PKG-INFO +93 -0
  3. webpage_parser-0.1.0/README_PYPI.md +83 -0
  4. webpage_parser-0.1.0/__init__.py +11 -0
  5. webpage_parser-0.1.0/author.py +503 -0
  6. webpage_parser-0.1.0/content.py +915 -0
  7. webpage_parser-0.1.0/extract.py +63 -0
  8. webpage_parser-0.1.0/htmlprep.py +760 -0
  9. webpage_parser-0.1.0/pagegate.py +177 -0
  10. webpage_parser-0.1.0/pipeline.py +196 -0
  11. webpage_parser-0.1.0/pubtime.py +490 -0
  12. webpage_parser-0.1.0/pyproject.toml +28 -0
  13. webpage_parser-0.1.0/requirements.txt +3 -0
  14. webpage_parser-0.1.0/setup.cfg +4 -0
  15. webpage_parser-0.1.0/siteconfig.py +110 -0
  16. webpage_parser-0.1.0/sitespecific.py +170 -0
  17. webpage_parser-0.1.0/timetext.py +365 -0
  18. webpage_parser-0.1.0/title.py +222 -0
  19. webpage_parser-0.1.0/util.py +95 -0
  20. webpage_parser-0.1.0/webpage_parser.egg-info/PKG-INFO +93 -0
  21. webpage_parser-0.1.0/webpage_parser.egg-info/SOURCES.txt +36 -0
  22. webpage_parser-0.1.0/webpage_parser.egg-info/dependency_links.txt +1 -0
  23. webpage_parser-0.1.0/webpage_parser.egg-info/entry_points.txt +2 -0
  24. webpage_parser-0.1.0/webpage_parser.egg-info/requires.txt +3 -0
  25. webpage_parser-0.1.0/webpage_parser.egg-info/top_level.txt +1 -0
webpage_parser-0.1.0/MANIFEST.in ADDED
@@ -0,0 +1,8 @@
1
+ # sdist 只带运行所需文件：requirements.txt 是 dynamic dependencies 的真相源，
2
+ # 不显式 include 的话从 sdist 重新构建会失败。
3
+ include requirements.txt
4
+ include README_PYPI.md
5
+ exclude README.md
6
+ prune deploy
7
+ prune examples
8
+ global-exclude __pycache__/* *.py[cod] .DS_Store
webpage_parser-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,93 @@
1
+ Metadata-Version: 2.4
2
+ Name: webpage-parser
3
+ Version: 0.1.0
4
+ Summary: Extract title, content, publication time and author from raw HTML pages
5
+ Requires-Python: >=3.10
6
+ Description-Content-Type: text/markdown
7
+ Requires-Dist: lxml==6.1.2
8
+ Requires-Dist: trafilatura==2.2.0
9
+ Requires-Dist: dateparser==1.4.2
10
+
11
+ # webpage-parser
12
+
13
+ Extract four core fields from a raw HTML page: `title`, `content`, `pub_time`, `author`.
14
+
15
+ Rule-based extraction over the DOM, using `lxml` for parsing, `trafilatura` for
16
+ main-content candidates and `dateparser` for multilingual date fallback.
17
+ No network access is performed while parsing.
18
+
19
+ ## Install
20
+
21
+ ```bash
22
+ pip install webpage-parser
23
+ ```
24
+
25
+ Requires Python >= 3.10 (`trafilatura` 2.2 and `dateparser` 1.4 both require it).
26
+
27
+ ## Python API
28
+
29
+ ```python
30
+ from webpage_parser import process_row
31
+
32
+ result = process_row({
33
+ "html": "<html>...</html>", # required
34
+ "url": "https://example.com/a/b", # optional, enables host-specific rules
35
+ "time": "2026-01-01 00:00:00", # optional, fetch time; upper bound for pub_time
36
+ "headers": {}, # optional dict
37
+ "extra": {}, # optional dict
38
+ })
39
+ ```
40
+
41
+ Only `html` is required. The other keys refine the result: `url` selects
42
+ host-specific rules, `time` rejects publication dates later than the fetch time.
43
+
44
+ Returns a `dict`. The four core fields are always present and always strings --
45
+ an empty string when nothing was extracted, never `None`:
46
+
47
+ | key | meaning |
48
+ |---|---|
49
+ | `title` | page title |
50
+ | `content` | main body text |
51
+ | `pub_time` | publication time, formatted `YYYY-MM-DD HH:MM:SS` |
52
+ | `author` | author or publishing organisation |
53
+
54
+ Additional diagnostic keys describe how each field was obtained:
55
+ `page_status`, `title_source`, `title_suffix_stripped`, `content_path`,
56
+ `content_traceable_ratio`, `content_noise_by_class`, `pub_time_source`,
57
+ `pub_time_precision`, `pub_time_tz_offset`, `pub_time_tz_inferred`,
58
+ `pub_time_raw`, `author_source`, `author_is_org`.
59
+
60
+ `page_status` is one of `ok`, `no_article`, `list_page`, `spa_empty`, `non_html`.
61
+ Body text is only extracted from pages that reach `ok`; thin, list-like or
62
+ client-rendered pages report the corresponding status and leave `content` empty.
63
+
64
+ `empty_result()` is also exported and returns the baseline dict with every field
65
+ empty. `process_row` may add `pub_time_tz_inferred` on top of those keys, so treat
66
+ the returned key set as a superset rather than a fixed schema.
67
+
68
+ ## Time zone
69
+
70
+ `pub_time` is normalised to **UTC+8**, the default assumed site time zone:
71
+
72
+ - a value carrying an explicit offset (`Z`, `+HH:MM`, `+HHMM`) is converted to
73
+ UTC+8, and `pub_time_tz_offset` keeps the original offset in minutes
74
+ - a value without any offset is interpreted using `extra["time_zone"]`, an offset
75
+ in hours (for example `0` or `-5`), falling back to UTC+8 when absent
76
+ - day-precision values are not shifted
77
+
78
+ ## Command line
79
+
80
+ ```bash
81
+ webpage-parser --input rows.jsonl --output preds.jsonl
82
+ ```
83
+
84
+ - one JSON object per input line, same keys as `process_row`
85
+ - output lines align 1:1 with input lines, in the same order
86
+ - a malformed or failing row degrades to empty values instead of aborting;
87
+ the exit code is always 0
88
+ - streaming read and write, the input is never loaded as a whole
89
+
90
+ ## Behaviour notes
91
+
92
+ - blank or non-string `html` yields `page_status = "non_html"`
93
+ - `html` longer than 8,000,000 characters is truncated before parsing
webpage_parser-0.1.0/README_PYPI.md ADDED
@@ -0,0 +1,83 @@
1
+ # webpage-parser
2
+
3
+ Extract four core fields from a raw HTML page: `title`, `content`, `pub_time`, `author`.
4
+
5
+ Rule-based extraction over the DOM, using `lxml` for parsing, `trafilatura` for
6
+ main-content candidates and `dateparser` for multilingual date fallback.
7
+ No network access is performed while parsing.
8
+
9
+ ## Install
10
+
11
+ ```bash
12
+ pip install webpage-parser
13
+ ```
14
+
15
+ Requires Python >= 3.10 (`trafilatura` 2.2 and `dateparser` 1.4 both require it).
16
+
17
+ ## Python API
18
+
19
+ ```python
20
+ from webpage_parser import process_row
21
+
22
+ result = process_row({
23
+ "html": "<html>...</html>", # required
24
+ "url": "https://example.com/a/b", # optional, enables host-specific rules
25
+ "time": "2026-01-01 00:00:00", # optional, fetch time; upper bound for pub_time
26
+ "headers": {}, # optional dict
27
+ "extra": {}, # optional dict
28
+ })
29
+ ```
30
+
31
+ Only `html` is required. The other keys refine the result: `url` selects
32
+ host-specific rules, `time` rejects publication dates later than the fetch time.
33
+
34
+ Returns a `dict`. The four core fields are always present and always strings --
35
+ an empty string when nothing was extracted, never `None`:
36
+
37
+ | key | meaning |
38
+ |---|---|
39
+ | `title` | page title |
40
+ | `content` | main body text |
41
+ | `pub_time` | publication time, formatted `YYYY-MM-DD HH:MM:SS` |
42
+ | `author` | author or publishing organisation |
43
+
44
+ Additional diagnostic keys describe how each field was obtained:
45
+ `page_status`, `title_source`, `title_suffix_stripped`, `content_path`,
46
+ `content_traceable_ratio`, `content_noise_by_class`, `pub_time_source`,
47
+ `pub_time_precision`, `pub_time_tz_offset`, `pub_time_tz_inferred`,
48
+ `pub_time_raw`, `author_source`, `author_is_org`.
49
+
50
+ `page_status` is one of `ok`, `no_article`, `list_page`, `spa_empty`, `non_html`.
51
+ Body text is only extracted from pages that reach `ok`; thin, list-like or
52
+ client-rendered pages report the corresponding status and leave `content` empty.
53
+
54
+ `empty_result()` is also exported and returns the baseline dict with every field
55
+ empty. `process_row` may add `pub_time_tz_inferred` on top of those keys, so treat
56
+ the returned key set as a superset rather than a fixed schema.
57
+
58
+ ## Time zone
59
+
60
+ `pub_time` is normalised to **UTC+8**, the default assumed site time zone:
61
+
62
+ - a value carrying an explicit offset (`Z`, `+HH:MM`, `+HHMM`) is converted to
63
+ UTC+8, and `pub_time_tz_offset` keeps the original offset in minutes
64
+ - a value without any offset is interpreted using `extra["time_zone"]`, an offset
65
+ in hours (for example `0` or `-5`), falling back to UTC+8 when absent
66
+ - day-precision values are not shifted
67
+
68
+ ## Command line
69
+
70
+ ```bash
71
+ webpage-parser --input rows.jsonl --output preds.jsonl
72
+ ```
73
+
74
+ - one JSON object per input line, same keys as `process_row`
75
+ - output lines align 1:1 with input lines, in the same order
76
+ - a malformed or failing row degrades to empty values instead of aborting;
77
+ the exit code is always 0
78
+ - streaming read and write, the input is never loaded as a whole
79
+
80
+ ## Behaviour notes
81
+
82
+ - blank or non-string `html` yields `page_status = "non_html"`
83
+ - `html` longer than 8,000,000 characters is truncated before parsing
webpage_parser-0.1.0/__init__.py ADDED
@@ -0,0 +1,11 @@
1
+ """网页解析算子：从爬取到的 HTML 抽取 title/content/pub_time/author。
2
+
3
+ 包内模块沿用「相对导入优先、扁平导入兜底」的双模式写法，因此既可以按包导入
4
+ （`from webpage_parser import process_row`），也可以把目录直接摊平当脚本跑
5
+ （`python3 extract.py`）。推荐按包导入。
6
+ """
7
+ __version__ = "0.1.0"
8
+
9
+ from .pipeline import empty_result, process_row
10
+
11
+ __all__ = ["empty_result", "process_row", "__version__"]