html2rss 0.28.0 → 0.29.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +82 -25
- data/html2rss.gemspec +0 -1
- data/lib/html2rss/article.rb +1 -2
- data/lib/html2rss/auto_source/README.md +8 -8
- data/lib/html2rss/auto_source/cleanup.rb +92 -1
- data/lib/html2rss/auto_source/scraper/json_state/document_scanner.rb +3 -5
- data/lib/html2rss/auto_source/scraper/meta_oembed.rb +15 -5
- data/lib/html2rss/auto_source/scraper/microdata.rb +8 -6
- data/lib/html2rss/auto_source/scraper/native_feed.rb +5 -2
- data/lib/html2rss/auto_source/scraper/schema.rb +39 -21
- data/lib/html2rss/auto_source/scraper.rb +2 -3
- data/lib/html2rss/auto_source/segmenter/cluster.rb +2 -3
- data/lib/html2rss/auto_source/segmenter/primary_link.rb +1 -2
- data/lib/html2rss/auto_source/segmenter.rb +1 -2
- data/lib/html2rss/auto_source.rb +25 -11
- data/lib/html2rss/batch.rb +169 -0
- data/lib/html2rss/capture/README.md +15 -6
- data/lib/html2rss/capture.rb +158 -15
- data/lib/html2rss/channel.rb +1 -2
- data/lib/html2rss/cli/probe_view.rb +66 -0
- data/lib/html2rss/cli/render.rb +124 -0
- data/lib/html2rss/cli/validate.rb +91 -0
- data/lib/html2rss/cli.rb +318 -193
- data/lib/html2rss/config/request_headers.rb +4 -14
- data/lib/html2rss/config/schema.rb +7 -5
- data/lib/html2rss/config/validator.rb +1 -0
- data/lib/html2rss/config.rb +59 -22
- data/lib/html2rss/doctor/botasaurus.rb +130 -0
- data/lib/html2rss/feed_builder/rss.rb +1 -2
- data/lib/html2rss/feed_pipeline/README.md +5 -5
- data/lib/html2rss/feed_pipeline/auto_fallback.rb +29 -10
- data/lib/html2rss/feed_pipeline.rb +27 -16
- data/lib/html2rss/feed_resolution/README.md +11 -11
- data/lib/html2rss/feed_resolution.rb +6 -12
- data/lib/html2rss/html/article_extractor/heading_extractor.rb +24 -14
- data/lib/html2rss/html/article_extractor.rb +4 -9
- data/lib/html2rss/html/feed_link.rb +4 -2
- data/lib/html2rss/html/navigator.rb +4 -39
- data/lib/html2rss/html/probe.rb +82 -0
- data/lib/html2rss/html/sst_article_extractor.rb +1 -2
- data/lib/html2rss/link_destination/noise_policy.rb +1 -2
- data/lib/html2rss/link_destination/path_classifier.rb +20 -19
- data/lib/html2rss/mcp/README.md +80 -0
- data/lib/html2rss/mcp/contract.rb +135 -13
- data/lib/html2rss/mcp/outcome/playbook.rb +122 -0
- data/lib/html2rss/mcp/outcome.rb +112 -42
- data/lib/html2rss/mcp/runtime.rb +45 -0
- data/lib/html2rss/mcp/server/tools.rb +277 -0
- data/lib/html2rss/mcp/server.rb +91 -232
- data/lib/html2rss/mcp.rb +3 -0
- data/lib/html2rss/page_recon/README.md +69 -0
- data/lib/html2rss/page_recon/diagnostics.rb +211 -0
- data/lib/html2rss/page_recon.rb +56 -2
- data/lib/html2rss/recon.rb +252 -0
- data/lib/html2rss/request_service/blocked_surface.rb +1 -0
- data/lib/html2rss/request_service/faraday_strategy.rb +11 -7
- data/lib/html2rss/request_service/policy.rb +1 -2
- data/lib/html2rss/request_service/response.rb +3 -4
- data/lib/html2rss/request_service/strategy.rb +1 -2
- data/lib/html2rss/request_service.rb +2 -0
- data/lib/html2rss/request_session.rb +1 -2
- data/lib/html2rss/scrape_target.rb +1 -1
- data/lib/html2rss/selectors/post_processors/sanitize_html.rb +2 -4
- data/lib/html2rss/selectors/post_processors.rb +0 -1
- data/lib/html2rss/selectors.rb +33 -10
- data/lib/html2rss/sst/attrs.rb +1 -2
- data/lib/html2rss/sst/normalizer.rb +49 -11
- data/lib/html2rss/status.rb +3 -6
- data/lib/html2rss/syndication/README.md +6 -6
- data/lib/html2rss/syndication/discovery.rb +2 -4
- data/lib/html2rss/test/enhance_audit.rb +192 -0
- data/lib/html2rss/test.rb +458 -0
- data/lib/html2rss/url.rb +1 -2
- data/lib/html2rss/version.rb +1 -1
- data/lib/html2rss.rb +157 -35
- data/schema/html2rss-config.schema.json +9 -29
- metadata +17 -18
- data/lib/html2rss/mcp/inspect.rb +0 -138
- data/lib/html2rss/selectors/post_processors/html_to_markdown.rb +0 -59
|
@@ -33,13 +33,12 @@ module Html2rss
|
|
|
33
33
|
# @param articles [Array<Html2rss::Article>]
|
|
34
34
|
# @param surface_category [Symbol, nil]
|
|
35
35
|
# @return [Result]
|
|
36
|
-
# rubocop:disable Metrics/ParameterLists -- tournament kwargs stay co-located
|
|
36
|
+
# rubocop:disable-next Metrics/ParameterLists -- tournament kwargs stay co-located
|
|
37
37
|
def self.call(entry_url:, response:, session:, config:, articles:, surface_category: nil)
|
|
38
38
|
Runner.new(
|
|
39
39
|
entry_url:, response:, session:, config:, articles:, surface_category:
|
|
40
40
|
).call
|
|
41
41
|
end
|
|
42
|
-
# rubocop:enable Metrics/ParameterLists
|
|
43
42
|
|
|
44
43
|
##
|
|
45
44
|
# Outcome of {FeedResolution.try_apply!} for the auto-fallback chain.
|
|
@@ -60,7 +59,7 @@ module Html2rss
|
|
|
60
59
|
# @param budget [Html2rss::RequestService::Budget]
|
|
61
60
|
# @return [ApplyOutcome, nil] `:succeeded` sticky scrape target when retry extract yielded items;
|
|
62
61
|
# `nil` when no winner or retry was empty (entry scrape target kept; Diag.applied may still be true)
|
|
63
|
-
# rubocop:disable Metrics/ParameterLists -- orchestration kwargs stay co-located
|
|
62
|
+
# rubocop:disable-next Metrics/ParameterLists -- orchestration kwargs stay co-located
|
|
64
63
|
def self.try_apply!(pipeline:, config:, response:, session:, strategy:, resources:, articles:,
|
|
65
64
|
scrape_target:, state:, budget:)
|
|
66
65
|
Orchestrator.new(
|
|
@@ -68,7 +67,6 @@ module Html2rss
|
|
|
68
67
|
scrape_target:, state:, budget:
|
|
69
68
|
).call
|
|
70
69
|
end
|
|
71
|
-
# rubocop:enable Metrics/ParameterLists
|
|
72
70
|
|
|
73
71
|
# Tournament + optional retry orchestration for {FeedPipeline::AutoFallback}.
|
|
74
72
|
class Orchestrator
|
|
@@ -89,7 +87,7 @@ module Html2rss
|
|
|
89
87
|
# @param scrape_target [Html2rss::ScrapeTarget]
|
|
90
88
|
# @param state [Html2rss::FeedPipeline::AutoFallback::AttemptState]
|
|
91
89
|
# @param budget [Html2rss::RequestService::Budget]
|
|
92
|
-
# rubocop:disable Metrics/ParameterLists -- orchestration context stays co-located
|
|
90
|
+
# rubocop:disable-next Metrics/ParameterLists -- orchestration context stays co-located
|
|
93
91
|
def initialize(pipeline:, config:, response:, session:, strategy:, resources:, articles:,
|
|
94
92
|
scrape_target:, state:, budget:)
|
|
95
93
|
@pipeline = pipeline
|
|
@@ -103,11 +101,10 @@ module Html2rss
|
|
|
103
101
|
@state = state
|
|
104
102
|
@budget = budget
|
|
105
103
|
end
|
|
106
|
-
# rubocop:enable Metrics/ParameterLists
|
|
107
104
|
|
|
108
105
|
##
|
|
109
106
|
# @return [ApplyOutcome, nil]
|
|
110
|
-
# rubocop:disable Metrics/MethodLength -- eligibility + tournament + retry path
|
|
107
|
+
# rubocop:disable-next Metrics/MethodLength -- eligibility + tournament + retry path
|
|
111
108
|
def call
|
|
112
109
|
return unless eligible?
|
|
113
110
|
|
|
@@ -131,7 +128,6 @@ module Html2rss
|
|
|
131
128
|
Log.warn("FeedResolution: entry resolution retry failed (#{error.class})")
|
|
132
129
|
nil
|
|
133
130
|
end
|
|
134
|
-
# rubocop:enable Metrics/MethodLength
|
|
135
131
|
|
|
136
132
|
private
|
|
137
133
|
|
|
@@ -192,7 +188,7 @@ module Html2rss
|
|
|
192
188
|
# @param config [Html2rss::Config]
|
|
193
189
|
# @param articles [Array<Html2rss::Article>]
|
|
194
190
|
# @param surface_category [Symbol, nil]
|
|
195
|
-
# rubocop:disable Metrics/ParameterLists -- tournament context stays co-located
|
|
191
|
+
# rubocop:disable-next Metrics/ParameterLists -- tournament context stays co-located
|
|
196
192
|
def initialize(entry_url:, response:, session:, config:, articles:, surface_category: nil)
|
|
197
193
|
raise ArgumentError, 'articles must be an Array' unless articles.is_a?(Array)
|
|
198
194
|
|
|
@@ -203,11 +199,10 @@ module Html2rss
|
|
|
203
199
|
@articles = articles
|
|
204
200
|
@surface_category = surface_category
|
|
205
201
|
end
|
|
206
|
-
# rubocop:enable Metrics/ParameterLists
|
|
207
202
|
|
|
208
203
|
##
|
|
209
204
|
# @return [Result]
|
|
210
|
-
# rubocop:disable Metrics/AbcSize, Metrics/MethodLength -- policy → candidates → probe → select
|
|
205
|
+
# rubocop:disable-next Metrics/AbcSize, Metrics/MethodLength -- policy → candidates → probe → select
|
|
211
206
|
def call
|
|
212
207
|
return skip(:policy_skip) unless Policy.resolve?(
|
|
213
208
|
config:, articles:, surface_category:
|
|
@@ -226,7 +221,6 @@ module Html2rss
|
|
|
226
221
|
|
|
227
222
|
apply(winner, probe_count: scored.size)
|
|
228
223
|
end
|
|
229
|
-
# rubocop:enable Metrics/AbcSize, Metrics/MethodLength
|
|
230
224
|
|
|
231
225
|
private
|
|
232
226
|
|
|
@@ -19,26 +19,22 @@ module Html2rss
|
|
|
19
19
|
tags = article_tag.css(HEADING_TAGS.join(','))
|
|
20
20
|
if tags.any?
|
|
21
21
|
select_best_heading(tags)
|
|
22
|
-
|
|
23
|
-
|
|
22
|
+
else
|
|
23
|
+
labeled = heading_from_aria_or_title(article_tag)
|
|
24
|
+
return labeled if labeled
|
|
25
|
+
|
|
26
|
+
fallback_heading(article_tag) if fallback_anchorless && selected_anchor.nil?
|
|
24
27
|
end
|
|
25
28
|
end
|
|
26
29
|
|
|
27
30
|
private
|
|
28
31
|
|
|
29
32
|
def select_best_heading(tags)
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
next if tag.name != min_tag_name
|
|
36
|
-
|
|
37
|
-
size = Navigator::TextExtractor.call(tag)&.size.to_i
|
|
38
|
-
(best_tag = tag) && (max_size = size) if size > max_size
|
|
39
|
-
end
|
|
40
|
-
|
|
41
|
-
best_tag
|
|
33
|
+
tagged = tags.map { |tag| [Probe.tag(tag), tag] }
|
|
34
|
+
min_tag_name = tagged.map(&:first).min
|
|
35
|
+
tagged.select { |name, _tag| name == min_tag_name }
|
|
36
|
+
.max_by { |_name, tag| Navigator::TextExtractor.call(tag)&.size.to_i }
|
|
37
|
+
&.last
|
|
42
38
|
end
|
|
43
39
|
|
|
44
40
|
def fallback_heading(article_tag)
|
|
@@ -47,6 +43,20 @@ module Html2rss
|
|
|
47
43
|
)
|
|
48
44
|
fallback_tags.find { |t| !Navigator::TextExtractor.call(t).to_s.strip.empty? }
|
|
49
45
|
end
|
|
46
|
+
|
|
47
|
+
def heading_from_aria_or_title(article_tag)
|
|
48
|
+
article_tag.css('[aria-label]').each do |node|
|
|
49
|
+
next if node['aria-label'].to_s.strip.empty?
|
|
50
|
+
|
|
51
|
+
return node
|
|
52
|
+
end
|
|
53
|
+
article_tag.css('[title]').each do |node|
|
|
54
|
+
next if node['title'].to_s.strip.empty?
|
|
55
|
+
|
|
56
|
+
return node
|
|
57
|
+
end
|
|
58
|
+
nil
|
|
59
|
+
end
|
|
50
60
|
end
|
|
51
61
|
end
|
|
52
62
|
end
|
|
@@ -5,7 +5,7 @@ module Html2rss
|
|
|
5
5
|
##
|
|
6
6
|
# ArticleExtractor is responsible for extracting details (headline, url, images, etc.)
|
|
7
7
|
# from an article_tag DOM node. DOM chrome helpers live on {Navigator}.
|
|
8
|
-
# rubocop:disable Metrics/ClassLength -- leftover re-extract stays with field extractors
|
|
8
|
+
# rubocop:disable-next Metrics/ClassLength -- leftover re-extract stays with field extractors
|
|
9
9
|
class ArticleExtractor
|
|
10
10
|
class << self
|
|
11
11
|
##
|
|
@@ -79,7 +79,7 @@ module Html2rss
|
|
|
79
79
|
Url.from_relative("##{id}", base_url) if id
|
|
80
80
|
end
|
|
81
81
|
|
|
82
|
-
# rubocop:disable Metrics/CyclomaticComplexity
|
|
82
|
+
# rubocop:disable-next Metrics/CyclomaticComplexity
|
|
83
83
|
def extract_title
|
|
84
84
|
source = heading || selected_anchor
|
|
85
85
|
title_text = source ? Navigator.extract_visible_text(source) : fallback_anchorless_title
|
|
@@ -88,7 +88,6 @@ module Html2rss
|
|
|
88
88
|
kicker = kicker_node ? Navigator.extract_visible_text(kicker_node).to_s.strip : nil
|
|
89
89
|
kicker && !kicker.empty? && !title_text.include?(kicker) ? "#{kicker}: #{title_text}" : title_text
|
|
90
90
|
end
|
|
91
|
-
# rubocop:enable Metrics/CyclomaticComplexity
|
|
92
91
|
|
|
93
92
|
def fallback_anchorless_title
|
|
94
93
|
return unless @fallback_anchorless && selected_anchor.nil?
|
|
@@ -130,11 +129,8 @@ module Html2rss
|
|
|
130
129
|
end
|
|
131
130
|
|
|
132
131
|
def heading_or_anchor_item?
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
def heading_item?
|
|
137
|
-
Navigator::HEADING_TAGS.include?(article_tag.name.to_s)
|
|
132
|
+
tag = Probe.tag(article_tag)
|
|
133
|
+
Navigator::HEADING_TAGS.include?(tag) || tag == 'a'
|
|
138
134
|
end
|
|
139
135
|
|
|
140
136
|
def heading_or_anchor_miss?(title, lines, published_at)
|
|
@@ -207,6 +203,5 @@ module Html2rss
|
|
|
207
203
|
|
|
208
204
|
def extract_enclosures = EnclosureExtractor.call(article_tag, base_url)
|
|
209
205
|
end
|
|
210
|
-
# rubocop:enable Metrics/ClassLength
|
|
211
206
|
end
|
|
212
207
|
end
|
|
@@ -25,8 +25,10 @@ module Html2rss
|
|
|
25
25
|
end
|
|
26
26
|
|
|
27
27
|
def feed_mime_type(type)
|
|
28
|
-
|
|
29
|
-
|
|
28
|
+
return Probe::APPLICATION_RSS_XML if Probe.mime_match?(type, Probe::APPLICATION_RSS_XML)
|
|
29
|
+
return Probe::APPLICATION_ATOM_XML if Probe.mime_match?(type, Probe::APPLICATION_ATOM_XML)
|
|
30
|
+
|
|
31
|
+
nil
|
|
30
32
|
end
|
|
31
33
|
end
|
|
32
34
|
end
|
|
@@ -11,9 +11,6 @@ module Html2rss
|
|
|
11
11
|
# Element tags that indicate ignored DOM chrome when found in a container path.
|
|
12
12
|
IGNORED_CONTAINER_TAGS = %w[nav footer header svg script style].to_set.freeze
|
|
13
13
|
|
|
14
|
-
# Layout roots and chrome tags excluded from class-clustering candidate nodes.
|
|
15
|
-
CLUSTER_EXCLUDED_TAGS = Set['html', 'body', 'nav', 'footer', 'header', 'svg', 'script', 'style'].freeze
|
|
16
|
-
|
|
17
14
|
# Ancestor tags that usually indicate navigation/utility regions inside a content container.
|
|
18
15
|
UTILITY_LANDMARK_TAGS = %w[nav aside footer menu].to_set.freeze
|
|
19
16
|
|
|
@@ -48,43 +45,11 @@ module Html2rss
|
|
|
48
45
|
# @param article_tag [Nokogiri::XML::Node] article-like container to search within
|
|
49
46
|
# @return [Nokogiri::XML::Node, nil] first eligible descendant anchor
|
|
50
47
|
def main_anchor_for(article_tag)
|
|
51
|
-
return article_tag if article_tag
|
|
48
|
+
return article_tag if Probe.tag(article_tag) == 'a' && article_tag.matches?(MAIN_ANCHOR_SELECTOR)
|
|
52
49
|
|
|
53
50
|
article_tag.at_css(MAIN_ANCHOR_SELECTOR)
|
|
54
51
|
end
|
|
55
52
|
|
|
56
|
-
##
|
|
57
|
-
# @param node [Nokogiri::XML::Node]
|
|
58
|
-
# @param cache [Hash, nil] identity cache used to store results (must use compare_by_identity)
|
|
59
|
-
# @return [Boolean] true when the node belongs to ignored DOM chrome
|
|
60
|
-
# rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
|
|
61
|
-
def ignored_container_path?(node, cache = nil)
|
|
62
|
-
return cache[node] if cache&.key?(node)
|
|
63
|
-
|
|
64
|
-
curr = node
|
|
65
|
-
visited = []
|
|
66
|
-
is_ignored = false
|
|
67
|
-
|
|
68
|
-
while curr.respond_to?(:parent) && curr
|
|
69
|
-
if cache&.key?(curr)
|
|
70
|
-
is_ignored = cache[curr]
|
|
71
|
-
break
|
|
72
|
-
end
|
|
73
|
-
|
|
74
|
-
if IGNORED_CONTAINER_TAGS.include?(curr.name)
|
|
75
|
-
is_ignored = true
|
|
76
|
-
break
|
|
77
|
-
end
|
|
78
|
-
|
|
79
|
-
visited << curr
|
|
80
|
-
curr = curr.parent
|
|
81
|
-
end
|
|
82
|
-
visited.each { |n| cache[n] = is_ignored } if cache
|
|
83
|
-
|
|
84
|
-
is_ignored
|
|
85
|
-
end
|
|
86
|
-
# rubocop:enable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
|
|
87
|
-
|
|
88
53
|
##
|
|
89
54
|
# Returns the first parent that satisfies the condition.
|
|
90
55
|
# If the condition is met, it returns the node itself.
|
|
@@ -93,7 +58,7 @@ module Html2rss
|
|
|
93
58
|
# @param condition [Proc] The condition to be met.
|
|
94
59
|
# @return [Nokogiri::XML::Node, nil] The first parent that satisfies the condition.
|
|
95
60
|
def parent_until_condition(node, condition)
|
|
96
|
-
while node && !node.document? && node
|
|
61
|
+
while node && !node.document? && Probe.tag(node) != 'html'
|
|
97
62
|
return node if condition.call(node)
|
|
98
63
|
|
|
99
64
|
node = node.parent
|
|
@@ -109,7 +74,7 @@ module Html2rss
|
|
|
109
74
|
return false unless node
|
|
110
75
|
return false if node.respond_to?(:document?) && node.document?
|
|
111
76
|
|
|
112
|
-
!CARD_WALK_STOP_TAGS.include?(node
|
|
77
|
+
!CARD_WALK_STOP_TAGS.include?(Probe.tag(node))
|
|
113
78
|
end
|
|
114
79
|
|
|
115
80
|
##
|
|
@@ -137,7 +102,7 @@ module Html2rss
|
|
|
137
102
|
# @param tag_name [String] tag name to find in ancestors
|
|
138
103
|
# @return [Nokogiri::XML::Node, nil] matching ancestor node
|
|
139
104
|
def find_tag_in_ancestors(current_tag, tag_name)
|
|
140
|
-
return current_tag if current_tag
|
|
105
|
+
return current_tag if Probe.tag(current_tag) == Probe.fold(tag_name)
|
|
141
106
|
|
|
142
107
|
current_tag.ancestors(tag_name).first
|
|
143
108
|
end
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Html2rss
|
|
4
|
+
module Html
|
|
5
|
+
##
|
|
6
|
+
# Case-insensitive DOM wire matching for MIME types, tag names, and link relations.
|
|
7
|
+
module Probe
|
|
8
|
+
# Canonical MIME type strings for script and alternate link matching.
|
|
9
|
+
APPLICATION_LD_JSON = 'application/ld+json'
|
|
10
|
+
# @see #mime_match?
|
|
11
|
+
APPLICATION_JSON = 'application/json'
|
|
12
|
+
# @see #mime_match?
|
|
13
|
+
APPLICATION_JSON_OEMBED = 'application/json+oembed'
|
|
14
|
+
# @see #mime_match?
|
|
15
|
+
APPLICATION_RSS_XML = 'application/rss+xml'
|
|
16
|
+
# @see #mime_match?
|
|
17
|
+
APPLICATION_ATOM_XML = 'application/atom+xml'
|
|
18
|
+
|
|
19
|
+
module_function
|
|
20
|
+
|
|
21
|
+
##
|
|
22
|
+
# @param string [String, Symbol, #to_s] wire value
|
|
23
|
+
# @return [String] folded comparison key
|
|
24
|
+
def fold(string)
|
|
25
|
+
string.to_s.downcase
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
##
|
|
29
|
+
# @param node [Nokogiri::XML::Node]
|
|
30
|
+
# @return [String] folded element name
|
|
31
|
+
def tag(node)
|
|
32
|
+
name = node.name
|
|
33
|
+
return fold(name) unless name.is_a?(String)
|
|
34
|
+
return name if name.ascii_only? && !name.match?(/[A-Z]/)
|
|
35
|
+
|
|
36
|
+
fold(name)
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
##
|
|
40
|
+
# @param type [String, Symbol, #to_s, nil] MIME type attribute value
|
|
41
|
+
# @return [String] media type without parameters, folded
|
|
42
|
+
def mime_base(type)
|
|
43
|
+
raw = type.to_s
|
|
44
|
+
return fold(raw) unless raw.include?(';') || raw.match?(/\A\s|\s\z/)
|
|
45
|
+
|
|
46
|
+
fold(raw.split(';', 2).first.to_s.strip)
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
##
|
|
50
|
+
# @param actual [String, Symbol, #to_s, nil] observed MIME type
|
|
51
|
+
# @param expected [Array<String>] canonical MIME types to match
|
|
52
|
+
# @return [Boolean]
|
|
53
|
+
def mime_match?(actual, *expected)
|
|
54
|
+
base = mime_base(actual)
|
|
55
|
+
expected.any? { |candidate| base == candidate || base == mime_base(candidate) }
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
##
|
|
59
|
+
# @param doc [Nokogiri::XML::Node]
|
|
60
|
+
# @param mime_types [Array<String>] optional MIME filters (folded once per call)
|
|
61
|
+
# @return [Array<Nokogiri::XML::Element>]
|
|
62
|
+
def scripts(doc, *mime_types)
|
|
63
|
+
return doc.css('script') if mime_types.empty?
|
|
64
|
+
|
|
65
|
+
allowed = mime_types.to_set { |mime| mime_base(mime) }
|
|
66
|
+
doc.css('script').select { |script| allowed.include?(mime_base(script['type'])) }
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
##
|
|
70
|
+
# @param doc [Nokogiri::XML::Node]
|
|
71
|
+
# @param rel [String] link rel token
|
|
72
|
+
# @param mime [String, nil] optional MIME filter
|
|
73
|
+
# @return [Array<Nokogiri::XML::Element>]
|
|
74
|
+
def alternate_links(doc, rel:, mime: nil)
|
|
75
|
+
nodes = doc.css(%(link[rel~="#{rel}"][href]))
|
|
76
|
+
return nodes if mime.nil?
|
|
77
|
+
|
|
78
|
+
nodes.select { |node| mime_match?(node['type'], mime) }
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
end
|
|
@@ -7,7 +7,7 @@ module Html2rss
|
|
|
7
7
|
##
|
|
8
8
|
# Builds an {Html2rss::Article} from an SST segment / ranked segment.
|
|
9
9
|
# Port of Html::ArticleExtractor field logic onto SST::Node.
|
|
10
|
-
# rubocop:disable Metrics/ClassLength -- SST field extractors colocated for parity with ArticleExtractor
|
|
10
|
+
# rubocop:disable-next Metrics/ClassLength -- SST field extractors colocated for parity with ArticleExtractor
|
|
11
11
|
class SstArticleExtractor
|
|
12
12
|
# CSS class tokens that mark kicker / eyebrow text (excluded from titles).
|
|
13
13
|
KICKER_CLASS_PATTERN = /kicker|eyebrow|pre-title|pretitle|overline/i
|
|
@@ -390,6 +390,5 @@ module Html2rss
|
|
|
390
390
|
end
|
|
391
391
|
end
|
|
392
392
|
end
|
|
393
|
-
# rubocop:enable Metrics/ClassLength
|
|
394
393
|
end
|
|
395
394
|
end
|
|
@@ -21,7 +21,7 @@ module Html2rss
|
|
|
21
21
|
# @param heading_anchor [Boolean]
|
|
22
22
|
# @param utility_landmark_ancestor [Boolean] pre-computed by Segmenter tree walk
|
|
23
23
|
# @return [Boolean]
|
|
24
|
-
# rubocop:disable Metrics/CyclomaticComplexity, Metrics/ParameterLists, Metrics/PerceivedComplexity, Lint/UnusedMethodArgument
|
|
24
|
+
# rubocop:disable-next Metrics/CyclomaticComplexity, Metrics/ParameterLists, Metrics/PerceivedComplexity, Lint/UnusedMethodArgument
|
|
25
25
|
def noise_anchor?(text:, destination_facts:, anchor: nil, container: nil,
|
|
26
26
|
heading_anchor: false, utility_landmark_ancestor: false)
|
|
27
27
|
return true unless destination_facts
|
|
@@ -35,7 +35,6 @@ module Html2rss
|
|
|
35
35
|
icon_only_anchor?(anchor, text) ||
|
|
36
36
|
utility_landmark_ancestor
|
|
37
37
|
end
|
|
38
|
-
# rubocop:enable Metrics/CyclomaticComplexity, Metrics/ParameterLists, Metrics/PerceivedComplexity, Lint/UnusedMethodArgument
|
|
39
38
|
|
|
40
39
|
private
|
|
41
40
|
|
|
@@ -71,34 +71,35 @@ module Html2rss
|
|
|
71
71
|
# @param segments [Array<String>] normalized URL path segments
|
|
72
72
|
def initialize(segments)
|
|
73
73
|
@segments = segments
|
|
74
|
+
@lexicon = segments.map { ::Html2rss::Html::Probe.fold(_1) }.freeze
|
|
74
75
|
end
|
|
75
76
|
|
|
76
77
|
# @return [Boolean] true when the route has article-like path evidence
|
|
77
78
|
def content_path?
|
|
78
79
|
@content_path ||= !leading_high_confidence_junk? &&
|
|
79
|
-
(SEGMENT_SETS[:content].intersect?(
|
|
80
|
+
(SEGMENT_SETS[:content].intersect?(@lexicon) || yearish_content_context?)
|
|
80
81
|
end
|
|
81
82
|
|
|
82
83
|
# @return [Boolean] true when the route includes utility/navigation evidence
|
|
83
84
|
def utility_path?
|
|
84
|
-
@utility_path ||= SEGMENT_SETS[:utility].intersect?(
|
|
85
|
+
@utility_path ||= SEGMENT_SETS[:utility].intersect?(@lexicon)
|
|
85
86
|
end
|
|
86
87
|
|
|
87
88
|
# @return [Boolean] true when the route points at conversion or account chrome
|
|
88
89
|
def vanity_path?
|
|
89
|
-
@vanity_path ||= SEGMENT_SETS[:vanity].intersect?(
|
|
90
|
+
@vanity_path ||= SEGMENT_SETS[:vanity].intersect?(@lexicon)
|
|
90
91
|
end
|
|
91
92
|
|
|
92
93
|
# @return [Boolean] true when the route points at taxonomy/listing chrome
|
|
93
94
|
def taxonomy_path?
|
|
94
|
-
@taxonomy_path ||= SEGMENT_SETS[:taxonomy].intersect?(
|
|
95
|
+
@taxonomy_path ||= SEGMENT_SETS[:taxonomy].intersect?(@lexicon)
|
|
95
96
|
end
|
|
96
97
|
|
|
97
98
|
# @return [Boolean] true when the route is too shallow to strongly indicate an article
|
|
98
99
|
def shallow?
|
|
99
100
|
segment_count = segments.size
|
|
100
101
|
|
|
101
|
-
segment_count <= 1 || (segment_count == 2 &&
|
|
102
|
+
segment_count <= 1 || (segment_count == 2 && high_confidence_junk_at?(segment_count - 1))
|
|
102
103
|
end
|
|
103
104
|
|
|
104
105
|
# @return [Boolean] true when the final path segment looks like a post slug
|
|
@@ -110,15 +111,15 @@ module Html2rss
|
|
|
110
111
|
|
|
111
112
|
# @return [Boolean] true when every path segment is utility chrome
|
|
112
113
|
def utility_only_route?
|
|
113
|
-
segments.all? { |
|
|
114
|
+
(0...segments.size).all? { |i| high_confidence_junk_at?(i) }
|
|
114
115
|
end
|
|
115
116
|
|
|
116
117
|
# @return [Boolean] true when the route is shallow and contains high-confidence noise
|
|
117
118
|
def shallow_high_confidence_route?
|
|
118
119
|
vanity_segments = SEGMENT_SETS.fetch(:vanity)
|
|
119
120
|
|
|
120
|
-
shallow? && segments.any? do |
|
|
121
|
-
|
|
121
|
+
shallow? && (0...segments.size).any? do |i|
|
|
122
|
+
high_confidence_junk_at?(i) || vanity_segments.include?(@lexicon[i])
|
|
122
123
|
end
|
|
123
124
|
end
|
|
124
125
|
|
|
@@ -148,11 +149,11 @@ module Html2rss
|
|
|
148
149
|
private
|
|
149
150
|
|
|
150
151
|
def leading_high_confidence_junk?
|
|
151
|
-
segments.any? &&
|
|
152
|
+
segments.any? && high_confidence_junk_at?(0)
|
|
152
153
|
end
|
|
153
154
|
|
|
154
155
|
def any_high_confidence_junk_segment?
|
|
155
|
-
segments.any? { |
|
|
156
|
+
(0...segments.size).any? { |i| high_confidence_junk_at?(i) }
|
|
156
157
|
end
|
|
157
158
|
|
|
158
159
|
def yearish_content_context?
|
|
@@ -178,11 +179,12 @@ module Html2rss
|
|
|
178
179
|
def all_junk?(limit)
|
|
179
180
|
return false if limit <= 0
|
|
180
181
|
|
|
181
|
-
(0...limit).all? { |i|
|
|
182
|
+
(0...limit).all? { |i| high_confidence_junk_at?(i) }
|
|
182
183
|
end
|
|
183
184
|
|
|
184
|
-
def
|
|
185
|
-
SEGMENT_SETS.fetch(:high_confidence_junk).include?(
|
|
185
|
+
def high_confidence_junk_at?(index)
|
|
186
|
+
SEGMENT_SETS.fetch(:high_confidence_junk).include?(@lexicon[index]) ||
|
|
187
|
+
host_shaped_segment?(segments[index])
|
|
186
188
|
end
|
|
187
189
|
|
|
188
190
|
def host_shaped_segment?(segment)
|
|
@@ -197,10 +199,9 @@ module Html2rss
|
|
|
197
199
|
context_segments = SEGMENT_SETS.fetch(:deep_post_context)
|
|
198
200
|
|
|
199
201
|
(0...limit).any? do |i|
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
context_segments.include?(segment)
|
|
202
|
+
content_segments.include?(@lexicon[i]) ||
|
|
203
|
+
segments[i].match?(PathClassifier::YEARISH_SEGMENT) ||
|
|
204
|
+
context_segments.include?(@lexicon[i])
|
|
204
205
|
end
|
|
205
206
|
end
|
|
206
207
|
|
|
@@ -209,8 +210,8 @@ module Html2rss
|
|
|
209
210
|
end
|
|
210
211
|
|
|
211
212
|
def excluded_last_segment?
|
|
212
|
-
|
|
213
|
-
|
|
213
|
+
last_i = segments.size - 1
|
|
214
|
+
high_confidence_junk_at?(last_i) || SEGMENT_SETS[:vanity].include?(@lexicon[last_i])
|
|
214
215
|
end
|
|
215
216
|
|
|
216
217
|
def slug_last_segment?
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# MCP — curation wire surface
|
|
2
|
+
|
|
3
|
+
MCP exposes the same seven user-facing verbs as the CLI and gem facades. Full contract and ownership: `CONTEXT.md` § Frozen contract. Contributor law: `AGENTS.md` § Curation CLI / MCP.
|
|
4
|
+
|
|
5
|
+
## Verb table
|
|
6
|
+
|
|
7
|
+
| Verb | MCP tool | Job |
|
|
8
|
+
| -------- | ---------- | ---------------------------------------------------- |
|
|
9
|
+
| inspect | `inspect` | Diagnostics (final URL, status, alternates, surface) |
|
|
10
|
+
| recon | `recon` | Verdict + native_feed preference |
|
|
11
|
+
| capture | `capture` | YAML draft config |
|
|
12
|
+
| validate | `validate` | Schema only |
|
|
13
|
+
| test | `test` | Schema + live extraction |
|
|
14
|
+
| apply | `apply` | Ship RSS from config |
|
|
15
|
+
| scrape | `scrape` | Articles now (one-shot) |
|
|
16
|
+
|
|
17
|
+
Batch: `batch_inspect`, `batch_recon`, `batch_scrape`.
|
|
18
|
+
|
|
19
|
+
**Golden path:** optional inspect → recon → capture → test → apply. Side door: validate. One-shot: scrape.
|
|
20
|
+
|
|
21
|
+
## Decision tree
|
|
22
|
+
|
|
23
|
+
1. **Articles now (no saved config)?** → `scrape` (or `batch_scrape` for multiple URLs). `strategy: "auto"` runs Faraday → Botasaurus; do not retry with explicit `faraday` after `auto`.
|
|
24
|
+
2. **Reusable feed YAML?** → `capture` → `test` → `apply`. `capture` returns YAML in `payload.yaml`. Strive `enhance: true` (false only when chrome leaks). `test` runs schema + live extraction; optional `compare_enhance` compares enhance off vs on. `apply` is the ship gate (`isError` on zero items). Both `test` and `apply` may include `quality_report.enhance_gains` when `selectors.items.enhance` is true.
|
|
25
|
+
3. **Weak scrape/capture or recon?** → `inspect` (or `batch_inspect`). When alternates warrant it, follow `next_step` to `recon`.
|
|
26
|
+
4. **Config already in hand?** → `validate` (schema only) → `test` → `apply`.
|
|
27
|
+
|
|
28
|
+
Prefer `capture` for durable configs; `scrape` / `batch_scrape` for one-shot extraction. Follow envelope `next_step` and `guidance`; do not parse scrape text as a raw item array.
|
|
29
|
+
|
|
30
|
+
## Envelope
|
|
31
|
+
|
|
32
|
+
Every tool result is one JSON object (text body and `structuredContent`):
|
|
33
|
+
|
|
34
|
+
| Field | Role |
|
|
35
|
+
| ----------- | ------------------------------------------------------------ |
|
|
36
|
+
| `ok` | Success vs schema/ship failure |
|
|
37
|
+
| `next_step` | Bare verb name or `done` / `read_runtime` |
|
|
38
|
+
| `guidance` | Agent instruction (from `Outcome::Playbook`) |
|
|
39
|
+
| `payload` | Tool-specific data (`yaml`, `rss`, `items`, recon fields, …) |
|
|
40
|
+
|
|
41
|
+
`Contract.response` builds the wire body; `Outcome` owns `next_step` policy.
|
|
42
|
+
|
|
43
|
+
## Ownership
|
|
44
|
+
|
|
45
|
+
| Concern | Owner |
|
|
46
|
+
| --------------------------------------------- | ------------------------------------------------ |
|
|
47
|
+
| Tool schemas, titles, strategy enum | `MCP::Contract` |
|
|
48
|
+
| Catalog fingerprint + `mcp_contract_version` | `MCP::Contract` (+ `Runtime.snapshot` wire) |
|
|
49
|
+
| Envelope factories, `next_step` routing | `MCP::Outcome` |
|
|
50
|
+
| Runtime instructions, guidance, prompt bodies | `Outcome::Playbook` (SSOT — `Server` delegates) |
|
|
51
|
+
| Diagnostic fetch + assess | `PageRecon::Diagnostics` |
|
|
52
|
+
| Curation verdict | `Recon` |
|
|
53
|
+
| Capture YAML product | `Capture::CaptureResult#yaml` |
|
|
54
|
+
| Batch concurrency | `Batch.map` (Thread pool; preserves input order) |
|
|
55
|
+
|
|
56
|
+
Do not duplicate playbook prose in `server.rb`.
|
|
57
|
+
|
|
58
|
+
## Resources
|
|
59
|
+
|
|
60
|
+
| URI | Description |
|
|
61
|
+
| ----------------------- | ---------------------------------------------------------- |
|
|
62
|
+
| `html2rss://schema` | Full JSON Schema for feed configurations |
|
|
63
|
+
| `html2rss://extractors` | Registered extractor names |
|
|
64
|
+
| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`) |
|
|
65
|
+
| `html2rss://runtime` | `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, `botasaurus_configured` (never the scraper URL). Fingerprint covers tool names, required keys, and `oneOf` branches; bump `mcp_contract_version` for envelope semantics. |
|
|
66
|
+
|
|
67
|
+
## Prompts
|
|
68
|
+
|
|
69
|
+
| Name | Description |
|
|
70
|
+
| --------------------- | ------------------------------------------------------------ |
|
|
71
|
+
| `scrape-webpage` | One `scrape` call; `inspect` only if weak or recon needed |
|
|
72
|
+
| `capture-feed-config` | Capture YAML → test → apply; catalog rewrite; strive enhance |
|
|
73
|
+
|
|
74
|
+
## Strategy note
|
|
75
|
+
|
|
76
|
+
`scrape` / `capture` with `strategy: "auto"` run the full AutoFallback chain. `inspect` maps `auto` to Faraday for cheap diagnostics; pin `botasaurus` when you need browser rendering for inspect.
|
|
77
|
+
|
|
78
|
+
## Inspect redirects
|
|
79
|
+
|
|
80
|
+
`payload.final_url` is the post-redirect landing URL. When it differs from the URL you passed and `status` is 4xx, inspect still followed redirects — retry on `final_url` or pass the site's canonical hostname (often `www`). Cross-host redirects set `Host` per hop; html2rss does not pin the entry hostname. See [`page_recon/README.md`](../page_recon/README.md).
|