fetch_util 0.3.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/Rakefile +5 -0
- data/lib/fetch_util/assets/extract.js +1 -1
- data/lib/fetch_util/browser/site_stabilization/gitlab_repo.rb +32 -0
- data/lib/fetch_util/browser/site_stabilization.rb +2 -0
- data/lib/fetch_util/browser/stabilization/page_flow.rb +1 -0
- data/lib/fetch_util/cli.rb +1 -0
- data/lib/fetch_util/fetcher.rb +545 -42
- data/lib/fetch_util/raw_docs_fallback.rb +40 -3
- data/lib/fetch_util/result.rb +100 -3
- data/lib/fetch_util/version.rb +1 -1
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8c0ea86360b3e6d6b5a5f9f04a670d57266e8af85d196a05fbe7bb377fac15da
|
|
4
|
+
data.tar.gz: 9ddba74537f2c152f0aadedb940a7827e2536ca89c0a026d92af4dd0bf828b07
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 17b3931b136825cc3d8a2e742532e94798a83a9616250e9b1790a8cb9c11923274d44594d3015eb9e72c05184b759d5170f4f3d02375c44291d2f19987cca89f
|
|
7
|
+
data.tar.gz: b2f473b462ae5ee530f27bb4325551d30eb7a582b0558b2479d1fe0f1d18f67aa0ae70adf14a66a7b45248c845b370eb6b8d4087bea55815354f47dd46030239
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,18 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## v0.3.1 - 2026-07-06
|
|
6
|
+
|
|
7
|
+
- Improve primary content-root selection across long documents, article pages, docs, Drupal and institutional pages, publisher abstracts, Substack, Discourse, Trope Wiki, PLOS, Elsevier, ACS, HighWire, and legal/statute pages so body content is preferred over chrome, teasers, and related rails.
|
|
8
|
+
- Expand list, index, card, and hub classification for section feeds, job results, government and legal portals, legal contents/search pages, institutional case lists, AEM/Drupal/custom CMS cards, commerce product grids, Statuspage boards, standards records, package registries, and public Mastodon timelines.
|
|
9
|
+
- Add or improve extraction for structured reference and scholarly records including arXiv abstracts, dictionary definitions, GitLab repository READMEs, Cornell Wex articles, legal conventions, US Code provisions, OEIS/RCSB/NIST records, NCBI/Ensembl gene records, IUCN species assessments, IEEE Xplore abstracts, and official statute text.
|
|
10
|
+
- Reduce false positive warnings for stale content, reposts, consent/paywall walls, Polish language mismatches, docs pages, subscriptions, scientific/legal/institutional pages, legal cases, DOI/publisher redirects, same-organization redirects, statutes, and treaty indexes.
|
|
11
|
+
- Improve error and interstitial handling for auth redirects, PDFs, dominant unavailable pages, soft errors, press-hold challenges, dead DOI redirects, legal soft 404s, publisher unavailable pages, not-found pages, nav-only records, short landing heroes, and privacy fragments.
|
|
12
|
+
- Clean markdown artifacts from malformed card grids, legal citations, table-based content, nested tables, treaty collections, article promo widgets, reference rails, undefined image text, and repeated list card details.
|
|
13
|
+
- Ensure browser extraction assets are rebuilt during verify, spec, build, and release paths; guard extraction asset manifest completeness.
|
|
14
|
+
- Consolidate shared extraction helpers for CMS component hubs, scholarly articles, platform profiles, repository READMEs, card lists, not-found/interstitial detection, legal text, table markdown, cleanup vocabulary, result field mapping, profile HTML/articles, Chrome stripping, platform signatures, list item results, and spec assertions.
|
|
15
|
+
- Ignore local Serena project metadata from Git.
|
|
16
|
+
|
|
5
17
|
## v0.3.0 - 2026-06-21
|
|
6
18
|
|
|
7
19
|
- Add generic portal, marketplace, and booking homepage root selection so lead-story lists are extracted from shell pages the previous docs, repo, and community dispatchers missed; already-handled homepages are unaffected.
|
data/Rakefile
CHANGED
|
@@ -15,4 +15,9 @@ task :verify_extract_assets do
|
|
|
15
15
|
ruby "script/build_extract_assets.rb", "--check"
|
|
16
16
|
end
|
|
17
17
|
|
|
18
|
+
task verify_extract_assets: :build_extract_assets
|
|
19
|
+
task spec: :build_extract_assets
|
|
20
|
+
task build: :build_extract_assets
|
|
21
|
+
task release: :build_extract_assets
|
|
22
|
+
|
|
18
23
|
task default: %i[verify_extract_assets spec rubocop]
|