html2rss 0.28.0 → 0.29.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +82 -25
  3. data/html2rss.gemspec +0 -1
  4. data/lib/html2rss/article.rb +1 -2
  5. data/lib/html2rss/auto_source/README.md +8 -8
  6. data/lib/html2rss/auto_source/cleanup.rb +92 -1
  7. data/lib/html2rss/auto_source/scraper/json_state/document_scanner.rb +3 -5
  8. data/lib/html2rss/auto_source/scraper/meta_oembed.rb +15 -5
  9. data/lib/html2rss/auto_source/scraper/microdata.rb +8 -6
  10. data/lib/html2rss/auto_source/scraper/native_feed.rb +5 -2
  11. data/lib/html2rss/auto_source/scraper/schema.rb +39 -21
  12. data/lib/html2rss/auto_source/scraper.rb +2 -3
  13. data/lib/html2rss/auto_source/segmenter/cluster.rb +2 -3
  14. data/lib/html2rss/auto_source/segmenter/primary_link.rb +1 -2
  15. data/lib/html2rss/auto_source/segmenter.rb +1 -2
  16. data/lib/html2rss/auto_source.rb +25 -11
  17. data/lib/html2rss/batch.rb +169 -0
  18. data/lib/html2rss/capture/README.md +15 -6
  19. data/lib/html2rss/capture.rb +158 -15
  20. data/lib/html2rss/channel.rb +1 -2
  21. data/lib/html2rss/cli/probe_view.rb +66 -0
  22. data/lib/html2rss/cli/render.rb +124 -0
  23. data/lib/html2rss/cli/validate.rb +91 -0
  24. data/lib/html2rss/cli.rb +318 -193
  25. data/lib/html2rss/config/request_headers.rb +4 -14
  26. data/lib/html2rss/config/schema.rb +7 -5
  27. data/lib/html2rss/config/validator.rb +1 -0
  28. data/lib/html2rss/config.rb +59 -22
  29. data/lib/html2rss/doctor/botasaurus.rb +130 -0
  30. data/lib/html2rss/feed_builder/rss.rb +1 -2
  31. data/lib/html2rss/feed_pipeline/README.md +5 -5
  32. data/lib/html2rss/feed_pipeline/auto_fallback.rb +29 -10
  33. data/lib/html2rss/feed_pipeline.rb +27 -16
  34. data/lib/html2rss/feed_resolution/README.md +11 -11
  35. data/lib/html2rss/feed_resolution.rb +6 -12
  36. data/lib/html2rss/html/article_extractor/heading_extractor.rb +24 -14
  37. data/lib/html2rss/html/article_extractor.rb +4 -9
  38. data/lib/html2rss/html/feed_link.rb +4 -2
  39. data/lib/html2rss/html/navigator.rb +4 -39
  40. data/lib/html2rss/html/probe.rb +82 -0
  41. data/lib/html2rss/html/sst_article_extractor.rb +1 -2
  42. data/lib/html2rss/link_destination/noise_policy.rb +1 -2
  43. data/lib/html2rss/link_destination/path_classifier.rb +20 -19
  44. data/lib/html2rss/mcp/README.md +80 -0
  45. data/lib/html2rss/mcp/contract.rb +135 -13
  46. data/lib/html2rss/mcp/outcome/playbook.rb +122 -0
  47. data/lib/html2rss/mcp/outcome.rb +112 -42
  48. data/lib/html2rss/mcp/runtime.rb +45 -0
  49. data/lib/html2rss/mcp/server/tools.rb +277 -0
  50. data/lib/html2rss/mcp/server.rb +91 -232
  51. data/lib/html2rss/mcp.rb +3 -0
  52. data/lib/html2rss/page_recon/README.md +69 -0
  53. data/lib/html2rss/page_recon/diagnostics.rb +211 -0
  54. data/lib/html2rss/page_recon.rb +56 -2
  55. data/lib/html2rss/recon.rb +252 -0
  56. data/lib/html2rss/request_service/blocked_surface.rb +1 -0
  57. data/lib/html2rss/request_service/faraday_strategy.rb +11 -7
  58. data/lib/html2rss/request_service/policy.rb +1 -2
  59. data/lib/html2rss/request_service/response.rb +3 -4
  60. data/lib/html2rss/request_service/strategy.rb +1 -2
  61. data/lib/html2rss/request_service.rb +2 -0
  62. data/lib/html2rss/request_session.rb +1 -2
  63. data/lib/html2rss/scrape_target.rb +1 -1
  64. data/lib/html2rss/selectors/post_processors/sanitize_html.rb +2 -4
  65. data/lib/html2rss/selectors/post_processors.rb +0 -1
  66. data/lib/html2rss/selectors.rb +33 -10
  67. data/lib/html2rss/sst/attrs.rb +1 -2
  68. data/lib/html2rss/sst/normalizer.rb +49 -11
  69. data/lib/html2rss/status.rb +3 -6
  70. data/lib/html2rss/syndication/README.md +6 -6
  71. data/lib/html2rss/syndication/discovery.rb +2 -4
  72. data/lib/html2rss/test/enhance_audit.rb +192 -0
  73. data/lib/html2rss/test.rb +458 -0
  74. data/lib/html2rss/url.rb +1 -2
  75. data/lib/html2rss/version.rb +1 -1
  76. data/lib/html2rss.rb +157 -35
  77. data/schema/html2rss-config.schema.json +9 -29
  78. metadata +17 -18
  79. data/lib/html2rss/mcp/inspect.rb +0 -138
  80. data/lib/html2rss/selectors/post_processors/html_to_markdown.rb +0 -59
@@ -33,13 +33,12 @@ module Html2rss
33
33
  # @param articles [Array<Html2rss::Article>]
34
34
  # @param surface_category [Symbol, nil]
35
35
  # @return [Result]
36
- # rubocop:disable Metrics/ParameterLists -- tournament kwargs stay co-located
36
+ # rubocop:disable-next Metrics/ParameterLists -- tournament kwargs stay co-located
37
37
  def self.call(entry_url:, response:, session:, config:, articles:, surface_category: nil)
38
38
  Runner.new(
39
39
  entry_url:, response:, session:, config:, articles:, surface_category:
40
40
  ).call
41
41
  end
42
- # rubocop:enable Metrics/ParameterLists
43
42
 
44
43
  ##
45
44
  # Outcome of {FeedResolution.try_apply!} for the auto-fallback chain.
@@ -60,7 +59,7 @@ module Html2rss
60
59
  # @param budget [Html2rss::RequestService::Budget]
61
60
  # @return [ApplyOutcome, nil] `:succeeded` sticky scrape target when retry extract yielded items;
62
61
  # `nil` when no winner or retry was empty (entry scrape target kept; Diag.applied may still be true)
63
- # rubocop:disable Metrics/ParameterLists -- orchestration kwargs stay co-located
62
+ # rubocop:disable-next Metrics/ParameterLists -- orchestration kwargs stay co-located
64
63
  def self.try_apply!(pipeline:, config:, response:, session:, strategy:, resources:, articles:,
65
64
  scrape_target:, state:, budget:)
66
65
  Orchestrator.new(
@@ -68,7 +67,6 @@ module Html2rss
68
67
  scrape_target:, state:, budget:
69
68
  ).call
70
69
  end
71
- # rubocop:enable Metrics/ParameterLists
72
70
 
73
71
  # Tournament + optional retry orchestration for {FeedPipeline::AutoFallback}.
74
72
  class Orchestrator
@@ -89,7 +87,7 @@ module Html2rss
89
87
  # @param scrape_target [Html2rss::ScrapeTarget]
90
88
  # @param state [Html2rss::FeedPipeline::AutoFallback::AttemptState]
91
89
  # @param budget [Html2rss::RequestService::Budget]
92
- # rubocop:disable Metrics/ParameterLists -- orchestration context stays co-located
90
+ # rubocop:disable-next Metrics/ParameterLists -- orchestration context stays co-located
93
91
  def initialize(pipeline:, config:, response:, session:, strategy:, resources:, articles:,
94
92
  scrape_target:, state:, budget:)
95
93
  @pipeline = pipeline
@@ -103,11 +101,10 @@ module Html2rss
103
101
  @state = state
104
102
  @budget = budget
105
103
  end
106
- # rubocop:enable Metrics/ParameterLists
107
104
 
108
105
  ##
109
106
  # @return [ApplyOutcome, nil]
110
- # rubocop:disable Metrics/MethodLength -- eligibility + tournament + retry path
107
+ # rubocop:disable-next Metrics/MethodLength -- eligibility + tournament + retry path
111
108
  def call
112
109
  return unless eligible?
113
110
 
@@ -131,7 +128,6 @@ module Html2rss
131
128
  Log.warn("FeedResolution: entry resolution retry failed (#{error.class})")
132
129
  nil
133
130
  end
134
- # rubocop:enable Metrics/MethodLength
135
131
 
136
132
  private
137
133
 
@@ -192,7 +188,7 @@ module Html2rss
192
188
  # @param config [Html2rss::Config]
193
189
  # @param articles [Array<Html2rss::Article>]
194
190
  # @param surface_category [Symbol, nil]
195
- # rubocop:disable Metrics/ParameterLists -- tournament context stays co-located
191
+ # rubocop:disable-next Metrics/ParameterLists -- tournament context stays co-located
196
192
  def initialize(entry_url:, response:, session:, config:, articles:, surface_category: nil)
197
193
  raise ArgumentError, 'articles must be an Array' unless articles.is_a?(Array)
198
194
 
@@ -203,11 +199,10 @@ module Html2rss
203
199
  @articles = articles
204
200
  @surface_category = surface_category
205
201
  end
206
- # rubocop:enable Metrics/ParameterLists
207
202
 
208
203
  ##
209
204
  # @return [Result]
210
- # rubocop:disable Metrics/AbcSize, Metrics/MethodLength -- policy → candidates → probe → select
205
+ # rubocop:disable-next Metrics/AbcSize, Metrics/MethodLength -- policy → candidates → probe → select
211
206
  def call
212
207
  return skip(:policy_skip) unless Policy.resolve?(
213
208
  config:, articles:, surface_category:
@@ -226,7 +221,6 @@ module Html2rss
226
221
 
227
222
  apply(winner, probe_count: scored.size)
228
223
  end
229
- # rubocop:enable Metrics/AbcSize, Metrics/MethodLength
230
224
 
231
225
  private
232
226
 
@@ -19,26 +19,22 @@ module Html2rss
19
19
  tags = article_tag.css(HEADING_TAGS.join(','))
20
20
  if tags.any?
21
21
  select_best_heading(tags)
22
- elsif fallback_anchorless && selected_anchor.nil?
23
- fallback_heading(article_tag)
22
+ else
23
+ labeled = heading_from_aria_or_title(article_tag)
24
+ return labeled if labeled
25
+
26
+ fallback_heading(article_tag) if fallback_anchorless && selected_anchor.nil?
24
27
  end
25
28
  end
26
29
 
27
30
  private
28
31
 
29
32
  def select_best_heading(tags)
30
- min_tag_name = tags.map(&:name).min
31
- best_tag = nil
32
- max_size = -1
33
-
34
- tags.each do |tag|
35
- next if tag.name != min_tag_name
36
-
37
- size = Navigator::TextExtractor.call(tag)&.size.to_i
38
- (best_tag = tag) && (max_size = size) if size > max_size
39
- end
40
-
41
- best_tag
33
+ tagged = tags.map { |tag| [Probe.tag(tag), tag] }
34
+ min_tag_name = tagged.map(&:first).min
35
+ tagged.select { |name, _tag| name == min_tag_name }
36
+ .max_by { |_name, tag| Navigator::TextExtractor.call(tag)&.size.to_i }
37
+ &.last
42
38
  end
43
39
 
44
40
  def fallback_heading(article_tag)
@@ -47,6 +43,20 @@ module Html2rss
47
43
  )
48
44
  fallback_tags.find { |t| !Navigator::TextExtractor.call(t).to_s.strip.empty? }
49
45
  end
46
+
47
+ def heading_from_aria_or_title(article_tag)
48
+ article_tag.css('[aria-label]').each do |node|
49
+ next if node['aria-label'].to_s.strip.empty?
50
+
51
+ return node
52
+ end
53
+ article_tag.css('[title]').each do |node|
54
+ next if node['title'].to_s.strip.empty?
55
+
56
+ return node
57
+ end
58
+ nil
59
+ end
50
60
  end
51
61
  end
52
62
  end
@@ -5,7 +5,7 @@ module Html2rss
5
5
  ##
6
6
  # ArticleExtractor is responsible for extracting details (headline, url, images, etc.)
7
7
  # from an article_tag DOM node. DOM chrome helpers live on {Navigator}.
8
- # rubocop:disable Metrics/ClassLength -- leftover re-extract stays with field extractors
8
+ # rubocop:disable-next Metrics/ClassLength -- leftover re-extract stays with field extractors
9
9
  class ArticleExtractor
10
10
  class << self
11
11
  ##
@@ -79,7 +79,7 @@ module Html2rss
79
79
  Url.from_relative("##{id}", base_url) if id
80
80
  end
81
81
 
82
- # rubocop:disable Metrics/CyclomaticComplexity
82
+ # rubocop:disable-next Metrics/CyclomaticComplexity
83
83
  def extract_title
84
84
  source = heading || selected_anchor
85
85
  title_text = source ? Navigator.extract_visible_text(source) : fallback_anchorless_title
@@ -88,7 +88,6 @@ module Html2rss
88
88
  kicker = kicker_node ? Navigator.extract_visible_text(kicker_node).to_s.strip : nil
89
89
  kicker && !kicker.empty? && !title_text.include?(kicker) ? "#{kicker}: #{title_text}" : title_text
90
90
  end
91
- # rubocop:enable Metrics/CyclomaticComplexity
92
91
 
93
92
  def fallback_anchorless_title
94
93
  return unless @fallback_anchorless && selected_anchor.nil?
@@ -130,11 +129,8 @@ module Html2rss
130
129
  end
131
130
 
132
131
  def heading_or_anchor_item?
133
- heading_item? || article_tag.name.to_s == 'a'
134
- end
135
-
136
- def heading_item?
137
- Navigator::HEADING_TAGS.include?(article_tag.name.to_s)
132
+ tag = Probe.tag(article_tag)
133
+ Navigator::HEADING_TAGS.include?(tag) || tag == 'a'
138
134
  end
139
135
 
140
136
  def heading_or_anchor_miss?(title, lines, published_at)
@@ -207,6 +203,5 @@ module Html2rss
207
203
 
208
204
  def extract_enclosures = EnclosureExtractor.call(article_tag, base_url)
209
205
  end
210
- # rubocop:enable Metrics/ClassLength
211
206
  end
212
207
  end
@@ -25,8 +25,10 @@ module Html2rss
25
25
  end
26
26
 
27
27
  def feed_mime_type(type)
28
- media_type = type.to_s.split(';', 2).first.strip.downcase
29
- media_type if media_type in 'application/rss+xml' | 'application/atom+xml'
28
+ return Probe::APPLICATION_RSS_XML if Probe.mime_match?(type, Probe::APPLICATION_RSS_XML)
29
+ return Probe::APPLICATION_ATOM_XML if Probe.mime_match?(type, Probe::APPLICATION_ATOM_XML)
30
+
31
+ nil
30
32
  end
31
33
  end
32
34
  end
@@ -11,9 +11,6 @@ module Html2rss
11
11
  # Element tags that indicate ignored DOM chrome when found in a container path.
12
12
  IGNORED_CONTAINER_TAGS = %w[nav footer header svg script style].to_set.freeze
13
13
 
14
- # Layout roots and chrome tags excluded from class-clustering candidate nodes.
15
- CLUSTER_EXCLUDED_TAGS = Set['html', 'body', 'nav', 'footer', 'header', 'svg', 'script', 'style'].freeze
16
-
17
14
  # Ancestor tags that usually indicate navigation/utility regions inside a content container.
18
15
  UTILITY_LANDMARK_TAGS = %w[nav aside footer menu].to_set.freeze
19
16
 
@@ -48,43 +45,11 @@ module Html2rss
48
45
  # @param article_tag [Nokogiri::XML::Node] article-like container to search within
49
46
  # @return [Nokogiri::XML::Node, nil] first eligible descendant anchor
50
47
  def main_anchor_for(article_tag)
51
- return article_tag if article_tag.name == 'a' && article_tag.matches?(MAIN_ANCHOR_SELECTOR)
48
+ return article_tag if Probe.tag(article_tag) == 'a' && article_tag.matches?(MAIN_ANCHOR_SELECTOR)
52
49
 
53
50
  article_tag.at_css(MAIN_ANCHOR_SELECTOR)
54
51
  end
55
52
 
56
- ##
57
- # @param node [Nokogiri::XML::Node]
58
- # @param cache [Hash, nil] identity cache used to store results (must use compare_by_identity)
59
- # @return [Boolean] true when the node belongs to ignored DOM chrome
60
- # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
61
- def ignored_container_path?(node, cache = nil)
62
- return cache[node] if cache&.key?(node)
63
-
64
- curr = node
65
- visited = []
66
- is_ignored = false
67
-
68
- while curr.respond_to?(:parent) && curr
69
- if cache&.key?(curr)
70
- is_ignored = cache[curr]
71
- break
72
- end
73
-
74
- if IGNORED_CONTAINER_TAGS.include?(curr.name)
75
- is_ignored = true
76
- break
77
- end
78
-
79
- visited << curr
80
- curr = curr.parent
81
- end
82
- visited.each { |n| cache[n] = is_ignored } if cache
83
-
84
- is_ignored
85
- end
86
- # rubocop:enable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
87
-
88
53
  ##
89
54
  # Returns the first parent that satisfies the condition.
90
55
  # If the condition is met, it returns the node itself.
@@ -93,7 +58,7 @@ module Html2rss
93
58
  # @param condition [Proc] The condition to be met.
94
59
  # @return [Nokogiri::XML::Node, nil] The first parent that satisfies the condition.
95
60
  def parent_until_condition(node, condition)
96
- while node && !node.document? && node.name != 'html'
61
+ while node && !node.document? && Probe.tag(node) != 'html'
97
62
  return node if condition.call(node)
98
63
 
99
64
  node = node.parent
@@ -109,7 +74,7 @@ module Html2rss
109
74
  return false unless node
110
75
  return false if node.respond_to?(:document?) && node.document?
111
76
 
112
- !CARD_WALK_STOP_TAGS.include?(node.name.to_s)
77
+ !CARD_WALK_STOP_TAGS.include?(Probe.tag(node))
113
78
  end
114
79
 
115
80
  ##
@@ -137,7 +102,7 @@ module Html2rss
137
102
  # @param tag_name [String] tag name to find in ancestors
138
103
  # @return [Nokogiri::XML::Node, nil] matching ancestor node
139
104
  def find_tag_in_ancestors(current_tag, tag_name)
140
- return current_tag if current_tag.name == tag_name
105
+ return current_tag if Probe.tag(current_tag) == Probe.fold(tag_name)
141
106
 
142
107
  current_tag.ancestors(tag_name).first
143
108
  end
@@ -0,0 +1,82 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Html2rss
4
+ module Html
5
+ ##
6
+ # Case-insensitive DOM wire matching for MIME types, tag names, and link relations.
7
+ module Probe
8
+ # Canonical MIME type strings for script and alternate link matching.
9
+ APPLICATION_LD_JSON = 'application/ld+json'
10
+ # @see #mime_match?
11
+ APPLICATION_JSON = 'application/json'
12
+ # @see #mime_match?
13
+ APPLICATION_JSON_OEMBED = 'application/json+oembed'
14
+ # @see #mime_match?
15
+ APPLICATION_RSS_XML = 'application/rss+xml'
16
+ # @see #mime_match?
17
+ APPLICATION_ATOM_XML = 'application/atom+xml'
18
+
19
+ module_function
20
+
21
+ ##
22
+ # @param string [String, Symbol, #to_s] wire value
23
+ # @return [String] folded comparison key
24
+ def fold(string)
25
+ string.to_s.downcase
26
+ end
27
+
28
+ ##
29
+ # @param node [Nokogiri::XML::Node]
30
+ # @return [String] folded element name
31
+ def tag(node)
32
+ name = node.name
33
+ return fold(name) unless name.is_a?(String)
34
+ return name if name.ascii_only? && !name.match?(/[A-Z]/)
35
+
36
+ fold(name)
37
+ end
38
+
39
+ ##
40
+ # @param type [String, Symbol, #to_s, nil] MIME type attribute value
41
+ # @return [String] media type without parameters, folded
42
+ def mime_base(type)
43
+ raw = type.to_s
44
+ return fold(raw) unless raw.include?(';') || raw.match?(/\A\s|\s\z/)
45
+
46
+ fold(raw.split(';', 2).first.to_s.strip)
47
+ end
48
+
49
+ ##
50
+ # @param actual [String, Symbol, #to_s, nil] observed MIME type
51
+ # @param expected [Array<String>] canonical MIME types to match
52
+ # @return [Boolean]
53
+ def mime_match?(actual, *expected)
54
+ base = mime_base(actual)
55
+ expected.any? { |candidate| base == candidate || base == mime_base(candidate) }
56
+ end
57
+
58
+ ##
59
+ # @param doc [Nokogiri::XML::Node]
60
+ # @param mime_types [Array<String>] optional MIME filters (folded once per call)
61
+ # @return [Array<Nokogiri::XML::Element>]
62
+ def scripts(doc, *mime_types)
63
+ return doc.css('script') if mime_types.empty?
64
+
65
+ allowed = mime_types.to_set { |mime| mime_base(mime) }
66
+ doc.css('script').select { |script| allowed.include?(mime_base(script['type'])) }
67
+ end
68
+
69
+ ##
70
+ # @param doc [Nokogiri::XML::Node]
71
+ # @param rel [String] link rel token
72
+ # @param mime [String, nil] optional MIME filter
73
+ # @return [Array<Nokogiri::XML::Element>]
74
+ def alternate_links(doc, rel:, mime: nil)
75
+ nodes = doc.css(%(link[rel~="#{rel}"][href]))
76
+ return nodes if mime.nil?
77
+
78
+ nodes.select { |node| mime_match?(node['type'], mime) }
79
+ end
80
+ end
81
+ end
82
+ end
@@ -7,7 +7,7 @@ module Html2rss
7
7
  ##
8
8
  # Builds an {Html2rss::Article} from an SST segment / ranked segment.
9
9
  # Port of Html::ArticleExtractor field logic onto SST::Node.
10
- # rubocop:disable Metrics/ClassLength -- SST field extractors colocated for parity with ArticleExtractor
10
+ # rubocop:disable-next Metrics/ClassLength -- SST field extractors colocated for parity with ArticleExtractor
11
11
  class SstArticleExtractor
12
12
  # CSS class tokens that mark kicker / eyebrow text (excluded from titles).
13
13
  KICKER_CLASS_PATTERN = /kicker|eyebrow|pre-title|pretitle|overline/i
@@ -390,6 +390,5 @@ module Html2rss
390
390
  end
391
391
  end
392
392
  end
393
- # rubocop:enable Metrics/ClassLength
394
393
  end
395
394
  end
@@ -21,7 +21,7 @@ module Html2rss
21
21
  # @param heading_anchor [Boolean]
22
22
  # @param utility_landmark_ancestor [Boolean] pre-computed by Segmenter tree walk
23
23
  # @return [Boolean]
24
- # rubocop:disable Metrics/CyclomaticComplexity, Metrics/ParameterLists, Metrics/PerceivedComplexity, Lint/UnusedMethodArgument
24
+ # rubocop:disable-next Metrics/CyclomaticComplexity, Metrics/ParameterLists, Metrics/PerceivedComplexity, Lint/UnusedMethodArgument
25
25
  def noise_anchor?(text:, destination_facts:, anchor: nil, container: nil,
26
26
  heading_anchor: false, utility_landmark_ancestor: false)
27
27
  return true unless destination_facts
@@ -35,7 +35,6 @@ module Html2rss
35
35
  icon_only_anchor?(anchor, text) ||
36
36
  utility_landmark_ancestor
37
37
  end
38
- # rubocop:enable Metrics/CyclomaticComplexity, Metrics/ParameterLists, Metrics/PerceivedComplexity, Lint/UnusedMethodArgument
39
38
 
40
39
  private
41
40
 
@@ -71,34 +71,35 @@ module Html2rss
71
71
  # @param segments [Array<String>] normalized URL path segments
72
72
  def initialize(segments)
73
73
  @segments = segments
74
+ @lexicon = segments.map { ::Html2rss::Html::Probe.fold(_1) }.freeze
74
75
  end
75
76
 
76
77
  # @return [Boolean] true when the route has article-like path evidence
77
78
  def content_path?
78
79
  @content_path ||= !leading_high_confidence_junk? &&
79
- (SEGMENT_SETS[:content].intersect?(segments) || yearish_content_context?)
80
+ (SEGMENT_SETS[:content].intersect?(@lexicon) || yearish_content_context?)
80
81
  end
81
82
 
82
83
  # @return [Boolean] true when the route includes utility/navigation evidence
83
84
  def utility_path?
84
- @utility_path ||= SEGMENT_SETS[:utility].intersect?(segments)
85
+ @utility_path ||= SEGMENT_SETS[:utility].intersect?(@lexicon)
85
86
  end
86
87
 
87
88
  # @return [Boolean] true when the route points at conversion or account chrome
88
89
  def vanity_path?
89
- @vanity_path ||= SEGMENT_SETS[:vanity].intersect?(segments)
90
+ @vanity_path ||= SEGMENT_SETS[:vanity].intersect?(@lexicon)
90
91
  end
91
92
 
92
93
  # @return [Boolean] true when the route points at taxonomy/listing chrome
93
94
  def taxonomy_path?
94
- @taxonomy_path ||= SEGMENT_SETS[:taxonomy].intersect?(segments)
95
+ @taxonomy_path ||= SEGMENT_SETS[:taxonomy].intersect?(@lexicon)
95
96
  end
96
97
 
97
98
  # @return [Boolean] true when the route is too shallow to strongly indicate an article
98
99
  def shallow?
99
100
  segment_count = segments.size
100
101
 
101
- segment_count <= 1 || (segment_count == 2 && high_confidence_junk_segment?(segments.last))
102
+ segment_count <= 1 || (segment_count == 2 && high_confidence_junk_at?(segment_count - 1))
102
103
  end
103
104
 
104
105
  # @return [Boolean] true when the final path segment looks like a post slug
@@ -110,15 +111,15 @@ module Html2rss
110
111
 
111
112
  # @return [Boolean] true when every path segment is utility chrome
112
113
  def utility_only_route?
113
- segments.all? { |segment| high_confidence_junk_segment?(segment) }
114
+ (0...segments.size).all? { |i| high_confidence_junk_at?(i) }
114
115
  end
115
116
 
116
117
  # @return [Boolean] true when the route is shallow and contains high-confidence noise
117
118
  def shallow_high_confidence_route?
118
119
  vanity_segments = SEGMENT_SETS.fetch(:vanity)
119
120
 
120
- shallow? && segments.any? do |segment|
121
- high_confidence_junk_segment?(segment) || vanity_segments.include?(segment)
121
+ shallow? && (0...segments.size).any? do |i|
122
+ high_confidence_junk_at?(i) || vanity_segments.include?(@lexicon[i])
122
123
  end
123
124
  end
124
125
 
@@ -148,11 +149,11 @@ module Html2rss
148
149
  private
149
150
 
150
151
  def leading_high_confidence_junk?
151
- segments.any? && high_confidence_junk_segment?(segments.first)
152
+ segments.any? && high_confidence_junk_at?(0)
152
153
  end
153
154
 
154
155
  def any_high_confidence_junk_segment?
155
- segments.any? { |segment| high_confidence_junk_segment?(segment) }
156
+ (0...segments.size).any? { |i| high_confidence_junk_at?(i) }
156
157
  end
157
158
 
158
159
  def yearish_content_context?
@@ -178,11 +179,12 @@ module Html2rss
178
179
  def all_junk?(limit)
179
180
  return false if limit <= 0
180
181
 
181
- (0...limit).all? { |i| high_confidence_junk_segment?(segments[i]) }
182
+ (0...limit).all? { |i| high_confidence_junk_at?(i) }
182
183
  end
183
184
 
184
- def high_confidence_junk_segment?(segment)
185
- SEGMENT_SETS.fetch(:high_confidence_junk).include?(segment) || host_shaped_segment?(segment)
185
+ def high_confidence_junk_at?(index)
186
+ SEGMENT_SETS.fetch(:high_confidence_junk).include?(@lexicon[index]) ||
187
+ host_shaped_segment?(segments[index])
186
188
  end
187
189
 
188
190
  def host_shaped_segment?(segment)
@@ -197,10 +199,9 @@ module Html2rss
197
199
  context_segments = SEGMENT_SETS.fetch(:deep_post_context)
198
200
 
199
201
  (0...limit).any? do |i|
200
- segment = segments[i]
201
- content_segments.include?(segment) ||
202
- segment.match?(PathClassifier::YEARISH_SEGMENT) ||
203
- context_segments.include?(segment)
202
+ content_segments.include?(@lexicon[i]) ||
203
+ segments[i].match?(PathClassifier::YEARISH_SEGMENT) ||
204
+ context_segments.include?(@lexicon[i])
204
205
  end
205
206
  end
206
207
 
@@ -209,8 +210,8 @@ module Html2rss
209
210
  end
210
211
 
211
212
  def excluded_last_segment?
212
- last = segments.last
213
- high_confidence_junk_segment?(last) || SEGMENT_SETS[:vanity].include?(last)
213
+ last_i = segments.size - 1
214
+ high_confidence_junk_at?(last_i) || SEGMENT_SETS[:vanity].include?(@lexicon[last_i])
214
215
  end
215
216
 
216
217
  def slug_last_segment?
@@ -0,0 +1,80 @@
1
+ # MCP — curation wire surface
2
+
3
+ MCP exposes the same seven user-facing verbs as the CLI and gem facades. Full contract and ownership: `CONTEXT.md` § Frozen contract. Contributor law: `AGENTS.md` § Curation CLI / MCP.
4
+
5
+ ## Verb table
6
+
7
+ | Verb | MCP tool | Job |
8
+ | -------- | ---------- | ---------------------------------------------------- |
9
+ | inspect | `inspect` | Diagnostics (final URL, status, alternates, surface) |
10
+ | recon | `recon` | Verdict + native_feed preference |
11
+ | capture | `capture` | YAML draft config |
12
+ | validate | `validate` | Schema only |
13
+ | test | `test` | Schema + live extraction |
14
+ | apply | `apply` | Ship RSS from config |
15
+ | scrape | `scrape` | Articles now (one-shot) |
16
+
17
+ Batch: `batch_inspect`, `batch_recon`, `batch_scrape`.
18
+
19
+ **Golden path:** optional inspect → recon → capture → test → apply. Side door: validate. One-shot: scrape.
20
+
21
+ ## Decision tree
22
+
23
+ 1. **Articles now (no saved config)?** → `scrape` (or `batch_scrape` for multiple URLs). `strategy: "auto"` runs Faraday → Botasaurus; do not retry with explicit `faraday` after `auto`.
24
+ 2. **Reusable feed YAML?** → `capture` → `test` → `apply`. `capture` returns YAML in `payload.yaml`. Strive `enhance: true` (false only when chrome leaks). `test` runs schema + live extraction; optional `compare_enhance` compares enhance off vs on. `apply` is the ship gate (`isError` on zero items). Both `test` and `apply` may include `quality_report.enhance_gains` when `selectors.items.enhance` is true.
25
+ 3. **Weak scrape/capture or recon?** → `inspect` (or `batch_inspect`). When alternates warrant it, follow `next_step` to `recon`.
26
+ 4. **Config already in hand?** → `validate` (schema only) → `test` → `apply`.
27
+
28
+ Prefer `capture` for durable configs; `scrape` / `batch_scrape` for one-shot extraction. Follow envelope `next_step` and `guidance`; do not parse scrape text as a raw item array.
29
+
30
+ ## Envelope
31
+
32
+ Every tool result is one JSON object (text body and `structuredContent`):
33
+
34
+ | Field | Role |
35
+ | ----------- | ------------------------------------------------------------ |
36
+ | `ok` | Success vs schema/ship failure |
37
+ | `next_step` | Bare verb name or `done` / `read_runtime` |
38
+ | `guidance` | Agent instruction (from `Outcome::Playbook`) |
39
+ | `payload` | Tool-specific data (`yaml`, `rss`, `items`, recon fields, …) |
40
+
41
+ `Contract.response` builds the wire body; `Outcome` owns `next_step` policy.
42
+
43
+ ## Ownership
44
+
45
+ | Concern | Owner |
46
+ | --------------------------------------------- | ------------------------------------------------ |
47
+ | Tool schemas, titles, strategy enum | `MCP::Contract` |
48
+ | Catalog fingerprint + `mcp_contract_version` | `MCP::Contract` (+ `Runtime.snapshot` wire) |
49
+ | Envelope factories, `next_step` routing | `MCP::Outcome` |
50
+ | Runtime instructions, guidance, prompt bodies | `Outcome::Playbook` (SSOT — `Server` delegates) |
51
+ | Diagnostic fetch + assess | `PageRecon::Diagnostics` |
52
+ | Curation verdict | `Recon` |
53
+ | Capture YAML product | `Capture::CaptureResult#yaml` |
54
+ | Batch concurrency | `Batch.map` (Thread pool; preserves input order) |
55
+
56
+ Do not duplicate playbook prose in `server.rb`.
57
+
58
+ ## Resources
59
+
60
+ | URI | Description |
61
+ | ----------------------- | ---------------------------------------------------------- |
62
+ | `html2rss://schema` | Full JSON Schema for feed configurations |
63
+ | `html2rss://extractors` | Registered extractor names |
64
+ | `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`) |
65
+ | `html2rss://runtime` | `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, `botasaurus_configured` (never the scraper URL). Fingerprint covers tool names, required keys, and `oneOf` branches; bump `mcp_contract_version` for envelope semantics. |
66
+
67
+ ## Prompts
68
+
69
+ | Name | Description |
70
+ | --------------------- | ------------------------------------------------------------ |
71
+ | `scrape-webpage` | One `scrape` call; `inspect` only if weak or recon needed |
72
+ | `capture-feed-config` | Capture YAML → test → apply; catalog rewrite; strive enhance |
73
+
74
+ ## Strategy note
75
+
76
+ `scrape` / `capture` with `strategy: "auto"` run the full AutoFallback chain. `inspect` maps `auto` to Faraday for cheap diagnostics; pin `botasaurus` when you need browser rendering for inspect.
77
+
78
+ ## Inspect redirects
79
+
80
+ `payload.final_url` is the post-redirect landing URL. When it differs from the URL you passed and `status` is 4xx, inspect still followed redirects — retry on `final_url` or pass the site's canonical hostname (often `www`). Cross-host redirects set `Host` per hop; html2rss does not pin the entry hostname. See [`page_recon/README.md`](../page_recon/README.md).