fetch_util 0.3.1 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +25 -0
- data/lib/fetch_util/assets/extract.js +1 -1
- data/lib/fetch_util/browser/interaction_helpers/consent_helpers.rb +187 -104
- data/lib/fetch_util/browser/interaction_helpers/dom_interaction.rb +14 -2
- data/lib/fetch_util/browser/navigation/headers_and_readiness.rb +0 -7
- data/lib/fetch_util/browser/navigation/navigator_patch.rb +3 -1
- data/lib/fetch_util/browser/site_stabilization/community_and_marketplace.rb +28 -27
- data/lib/fetch_util/browser/site_stabilization/social_platforms.rb +25 -32
- data/lib/fetch_util/browser/stabilization/page_flow.rb +93 -11
- data/lib/fetch_util/browser.rb +55 -16
- data/lib/fetch_util/extractor.rb +27 -3
- data/lib/fetch_util/fetcher.rb +270 -144
- data/lib/fetch_util/parallel_fetcher.rb +15 -20
- data/lib/fetch_util/raw_docs_fallback.rb +8 -21
- data/lib/fetch_util/regulatory/http_client.rb +111 -43
- data/lib/fetch_util/version.rb +1 -1
- data/lib/fetch_util.rb +12 -2
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8f8adcce7137bb744a361c30f0a2f47ea9f26e751cab745f0b298148151c0ea8
|
|
4
|
+
data.tar.gz: edbff86bc67288848d4e4c2548c6e8c5b7b53bcc4c29cc10a3d70e9831d19dec
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 85ce1b2472bb679c4f5941d3f828cfbc6455732b81c31952e1d9a5582fbfaa78af247e3ea88fe7a20fdfdcd60d6f50b124e662d6f139891c703060938ff4977d
|
|
7
|
+
data.tar.gz: b43a3c4ac0ab6d469c895ca8a12aca159b9d8112de63074497dec4e73178340b2bab666c5f2c7bb3a7ac9f1326c551816a16fed65733f06449a95bab1178a150
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,31 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## v0.3.2 - 2026-07-09
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- Add dedicated extraction coverage for public Chosun, Xinhua weekly indexes, O Pais, Avesta, Walla, Jang, 20minutos, NHK, Protothema, Lenta, and YNA article/list pages.
|
|
10
|
+
- Add a Hurriyet homepage regression fixture to protect non-English homepage extraction.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- Normalize non-ASCII IRI input URLs before browser navigation.
|
|
15
|
+
- Harden fetch resilience by retrying transient navigation and pending-connection failures.
|
|
16
|
+
- Improve generic article extraction and MediaWiki/Wikipedia handling while preserving localized truncation and warning semantics.
|
|
17
|
+
- Reduce false positives for homepage index mismatches, stale content, script-language mismatches, short news CTA multi-topic pages, transliterated slugs, and long non-English articles.
|
|
18
|
+
- Preserve article content through video metadata, France24 DOM stabilization, and short-description-root edge cases.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
- Consolidate list extraction helpers and reduce duplicated extraction/profile code across generic, WordPress, simple-profile, and job-list paths.
|
|
23
|
+
- Refine generic extraction and non-Latin spec style without changing the public API.
|
|
24
|
+
|
|
25
|
+
### Performance
|
|
26
|
+
|
|
27
|
+
- Speed up browser stabilization and Ruby fetch runtime.
|
|
28
|
+
- Share browser pools, avoid unnecessary asset rebuilds in specs, optimize asset checks/readable text, and streamline extractor integration specs.
|
|
29
|
+
|
|
5
30
|
## v0.3.1 - 2026-07-06
|
|
6
31
|
|
|
7
32
|
- Improve primary content-root selection across long documents, article pages, docs, Drupal and institutional pages, publisher abstracts, Substack, Discourse, Trope Wiki, PLOS, Elsevier, ACS, HighWire, and legal/statute pages so body content is preferred over chrome, teasers, and related rails.
|