html2rss 0.29.1 → 0.31.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +8 -8
- data/html2rss.gemspec +1 -3
- data/lib/html2rss/article/deduplicator.rb +2 -4
- data/lib/html2rss/article.rb +16 -39
- data/lib/html2rss/auto_source/README.md +1 -1
- data/lib/html2rss/auto_source/cleanup.rb +15 -6
- data/lib/html2rss/auto_source/scraper/html.rb +5 -0
- data/lib/html2rss/auto_source/scraper/json_state.rb +5 -0
- data/lib/html2rss/auto_source/scraper/meta_oembed.rb +8 -1
- data/lib/html2rss/auto_source/scraper/microdata.rb +8 -0
- data/lib/html2rss/auto_source/scraper/microformats2.rb +8 -0
- data/lib/html2rss/auto_source/scraper/native_feed.rb +3 -0
- data/lib/html2rss/auto_source/scraper/schema/thing.rb +4 -5
- data/lib/html2rss/auto_source/scraper/schema.rb +8 -0
- data/lib/html2rss/auto_source/scraper/semantic_html.rb +5 -0
- data/lib/html2rss/auto_source/scraper/sitemap.rb +3 -0
- data/lib/html2rss/auto_source/scraper/wordpress_api.rb +3 -0
- data/lib/html2rss/auto_source/scraper.rb +4 -14
- data/lib/html2rss/auto_source/segment.rb +7 -4
- data/lib/html2rss/auto_source/segmenter/cluster.rb +22 -23
- data/lib/html2rss/auto_source/segmenter/list.rb +54 -31
- data/lib/html2rss/auto_source/segmenter.rb +5 -5
- data/lib/html2rss/auto_source.rb +2 -2
- data/lib/html2rss/batch.rb +6 -4
- data/lib/html2rss/capture/README.md +7 -5
- data/lib/html2rss/capture.rb +68 -36
- data/lib/html2rss/channel.rb +3 -3
- data/lib/html2rss/cli/probe_view.rb +5 -3
- data/lib/html2rss/cli/render.rb +6 -5
- data/lib/html2rss/cli/validate.rb +28 -27
- data/lib/html2rss/cli.rb +1 -1
- data/lib/html2rss/config/issue_mapper.rb +108 -0
- data/lib/html2rss/config/request_headers.rb +6 -4
- data/lib/html2rss/config/schema.rb +16 -1
- data/lib/html2rss/config/selectors_validator.rb +92 -70
- data/lib/html2rss/config/validation_issue.rb +60 -0
- data/lib/html2rss/config/validation_report.rb +55 -0
- data/lib/html2rss/config/validator.rb +10 -6
- data/lib/html2rss/config.rb +35 -101
- data/lib/html2rss/defaults.rb +42 -30
- data/lib/html2rss/doctor/botasaurus.rb +19 -9
- data/lib/html2rss/doctor.rb +8 -0
- data/lib/html2rss/error.rb +1 -1
- data/lib/html2rss/feed_builder/rss.rb +9 -14
- data/lib/html2rss/feed_pipeline/README.md +8 -8
- data/lib/html2rss/feed_pipeline/auto_fallback.rb +1 -1
- data/lib/html2rss/feed_pipeline/runtime_policy.rb +14 -10
- data/lib/html2rss/feed_pipeline/strategy_plan.rb +7 -3
- data/lib/html2rss/feed_resolution/candidate_generator.rb +1 -4
- data/lib/html2rss/feed_resolution/diag.rb +3 -1
- data/lib/html2rss/feed_resolution/options.rb +3 -1
- data/lib/html2rss/feed_resolution/policy.rb +2 -2
- data/lib/html2rss/html/article_extractor/date_extractor.rb +1 -1
- data/lib/html2rss/html/article_extractor.rb +2 -4
- data/lib/html2rss/html/feed_link.rb +3 -1
- data/lib/html2rss/html/navigator/text_extractor.rb +1 -1
- data/lib/html2rss/html/navigator.rb +10 -10
- data/lib/html2rss/html/rendering/description_builder.rb +1 -1
- data/lib/html2rss/html/sst_article_extractor.rb +8 -5
- data/lib/html2rss/link_destination/destination_facts.rb +4 -2
- data/lib/html2rss/link_destination/noise_policy.rb +10 -1
- data/lib/html2rss/link_destination/path_classifier.rb +45 -38
- data/lib/html2rss/mcp/README.md +5 -3
- data/lib/html2rss/mcp/config_argument.rb +3 -1
- data/lib/html2rss/mcp/contract.rb +15 -25
- data/lib/html2rss/mcp/outcome/playbook.rb +7 -7
- data/lib/html2rss/mcp/outcome.rb +36 -43
- data/lib/html2rss/mcp/server/tools.rb +112 -132
- data/lib/html2rss/mcp/server.rb +27 -19
- data/lib/html2rss/page_recon/README.md +5 -5
- data/lib/html2rss/page_recon/diagnostics.rb +71 -81
- data/lib/html2rss/page_recon.rb +9 -5
- data/lib/html2rss/recon.rb +20 -31
- data/lib/html2rss/request_service/botasaurus_contract.rb +11 -12
- data/lib/html2rss/request_service/botasaurus_strategy.rb +32 -28
- data/lib/html2rss/request_service/compressed_body.rb +17 -9
- data/lib/html2rss/request_service/httpx_strategy.rb +239 -0
- data/lib/html2rss/request_service/network_guard.rb +7 -5
- data/lib/html2rss/request_service/response.rb +1 -2
- data/lib/html2rss/request_service/response_guard.rb +0 -16
- data/lib/html2rss/request_service/strategy.rb +22 -11
- data/lib/html2rss/request_service.rb +46 -85
- data/lib/html2rss/scoring/container_assessor.rb +3 -1
- data/lib/html2rss/scoring/engine.rb +8 -8
- data/lib/html2rss/scoring/observation.rb +5 -3
- data/lib/html2rss/scoring/ranked_segment.rb +3 -1
- data/lib/html2rss/scrape_target.rb +3 -1
- data/lib/html2rss/selector_candidates.rb +234 -0
- data/lib/html2rss/selectors/README.md +64 -0
- data/lib/html2rss/selectors/extractors/attribute.rb +13 -15
- data/lib/html2rss/selectors/extractors/href.rb +15 -16
- data/lib/html2rss/selectors/extractors/html.rb +8 -9
- data/lib/html2rss/selectors/extractors/static.rb +12 -14
- data/lib/html2rss/selectors/extractors/text.rb +8 -9
- data/lib/html2rss/selectors/extractors.rb +33 -20
- data/lib/html2rss/selectors/item_env.rb +31 -0
- data/lib/html2rss/selectors/{object_to_xml_converter.rb → json_xml.rb} +11 -12
- data/lib/html2rss/selectors/option_spec.rb +140 -0
- data/lib/html2rss/selectors/post_processors/base.rb +79 -31
- data/lib/html2rss/selectors/post_processors/gsub.rb +103 -39
- data/lib/html2rss/selectors/post_processors/html_transformers/transform_urls_to_absolute_ones.rb +4 -4
- data/lib/html2rss/selectors/post_processors/markdown_to_html.rb +10 -11
- data/lib/html2rss/selectors/post_processors/parse_time.rb +50 -26
- data/lib/html2rss/selectors/post_processors/parse_uri.rb +11 -7
- data/lib/html2rss/selectors/post_processors/sanitize_html.rb +46 -32
- data/lib/html2rss/selectors/post_processors/substring.rb +14 -25
- data/lib/html2rss/selectors/post_processors/template.rb +23 -19
- data/lib/html2rss/selectors/post_processors.rb +6 -5
- data/lib/html2rss/selectors/{schema_doc.rb → schema_export.rb} +12 -31
- data/lib/html2rss/selectors/step_env.rb +24 -0
- data/lib/html2rss/selectors.rb +231 -142
- data/lib/html2rss/sst/attrs.rb +5 -3
- data/lib/html2rss/sst/document.rb +3 -1
- data/lib/html2rss/sst/index.rb +3 -1
- data/lib/html2rss/sst/node.rb +29 -10
- data/lib/html2rss/sst/normalizer.rb +12 -16
- data/lib/html2rss/sst/tags.rb +7 -7
- data/lib/html2rss/status.rb +13 -8
- data/lib/html2rss/surface_category.rb +14 -32
- data/lib/html2rss/syndication/parser.rb +2 -2
- data/lib/html2rss/test/enhance_audit.rb +1 -3
- data/lib/html2rss/test/policy.rb +106 -0
- data/lib/html2rss/test.rb +126 -172
- data/lib/html2rss/url.rb +25 -4
- data/lib/html2rss/version.rb +1 -1
- data/lib/html2rss.rb +27 -19
- data/lib/tasks/config_schema.rake +2 -2
- data/schema/html2rss-config.schema.json +11 -8
- metadata +19 -44
- data/lib/html2rss/request_service/faraday_strategy.rb +0 -233
- data/lib/html2rss/selectors/item_scope.rb +0 -30
data/lib/html2rss/mcp/outcome.rb
CHANGED
|
@@ -13,24 +13,16 @@ module Html2rss
|
|
|
13
13
|
NextStep = Data.define(:name, :guidance)
|
|
14
14
|
|
|
15
15
|
##
|
|
16
|
-
# Closed set of agent next actions.
|
|
16
|
+
# Closed set of agent next actions. Names are owned by {Playbook::GUIDANCE}.
|
|
17
17
|
class NextStep
|
|
18
|
-
#
|
|
19
|
-
|
|
18
|
+
# Frozen name → instance table built from {Playbook::GUIDANCE}.
|
|
19
|
+
ALL = Playbook::GUIDANCE.to_h { |name, guidance| [name, new(name:, guidance:)] }.freeze
|
|
20
|
+
private_class_method :new
|
|
20
21
|
|
|
21
22
|
##
|
|
22
23
|
# @param name [Symbol, String]
|
|
23
|
-
# @
|
|
24
|
-
def
|
|
25
|
-
step = name.to_sym
|
|
26
|
-
raise ArgumentError, "unknown next_step: #{name.inspect}" unless NAMES.include?(step)
|
|
27
|
-
|
|
28
|
-
super(name: step, guidance: (guidance || Playbook::GUIDANCE.fetch(step)).freeze)
|
|
29
|
-
end
|
|
30
|
-
|
|
31
|
-
class << self
|
|
32
|
-
NAMES.each { |step| define_method(step) { new(name: step) } }
|
|
33
|
-
end
|
|
24
|
+
# @return [NextStep]
|
|
25
|
+
def self.[](name) = ALL.fetch(name.to_sym) { raise ArgumentError, "unknown next_step: #{name.inspect}" }
|
|
34
26
|
end
|
|
35
27
|
|
|
36
28
|
##
|
|
@@ -107,12 +99,13 @@ module Html2rss
|
|
|
107
99
|
end
|
|
108
100
|
|
|
109
101
|
##
|
|
110
|
-
# @param
|
|
102
|
+
# @param report [Html2rss::Config::ValidationReport]
|
|
111
103
|
# @return [Outcome]
|
|
112
|
-
def validate(
|
|
113
|
-
ok =
|
|
114
|
-
next_step = ok ? NextStep
|
|
115
|
-
|
|
104
|
+
def validate(report:)
|
|
105
|
+
ok = report.success?
|
|
106
|
+
next_step = ok ? NextStep[:test] : NextStep[:validate]
|
|
107
|
+
payload = ok ? {} : { issues: report.issues.map(&:to_h) }
|
|
108
|
+
new(ok:, next_step:, guidance: next_step.guidance, payload:)
|
|
116
109
|
end
|
|
117
110
|
|
|
118
111
|
##
|
|
@@ -131,17 +124,17 @@ module Html2rss
|
|
|
131
124
|
##
|
|
132
125
|
# @param batch_result [Html2rss::Batch::BatchResult]
|
|
133
126
|
# @return [Outcome]
|
|
134
|
-
def batch_scrape(batch_result) = batch(batch_result, NextStep
|
|
127
|
+
def batch_scrape(batch_result) = batch(batch_result, NextStep[:scrape])
|
|
135
128
|
|
|
136
129
|
##
|
|
137
130
|
# @param batch_result [Html2rss::Batch::BatchResult]
|
|
138
131
|
# @return [Outcome]
|
|
139
|
-
def batch_inspect(batch_result) = batch(batch_result, NextStep
|
|
132
|
+
def batch_inspect(batch_result) = batch(batch_result, NextStep[:inspect])
|
|
140
133
|
|
|
141
134
|
##
|
|
142
135
|
# @param batch_result [Html2rss::Batch::BatchResult]
|
|
143
136
|
# @return [Outcome]
|
|
144
|
-
def batch_recon(batch_result) = batch(batch_result, NextStep
|
|
137
|
+
def batch_recon(batch_result) = batch(batch_result, NextStep[:recon])
|
|
145
138
|
|
|
146
139
|
##
|
|
147
140
|
# @param rss [String]
|
|
@@ -151,7 +144,7 @@ module Html2rss
|
|
|
151
144
|
# @return [Outcome]
|
|
152
145
|
def apply(rss:, item_count:, empty: item_count.zero?, quality_report: nil)
|
|
153
146
|
ok = !empty
|
|
154
|
-
next_step = ok ? NextStep
|
|
147
|
+
next_step = ok ? NextStep[:done] : NextStep[:inspect]
|
|
155
148
|
payload = { rss:, item_count: }
|
|
156
149
|
payload[:quality_report] = quality_report if quality_report
|
|
157
150
|
new(ok:, next_step:, guidance: next_step.guidance, payload:)
|
|
@@ -169,15 +162,15 @@ module Html2rss
|
|
|
169
162
|
private
|
|
170
163
|
|
|
171
164
|
def batch(batch_result, failure_step)
|
|
172
|
-
step = batch_result.successful.positive? ? NextStep
|
|
165
|
+
step = batch_result.successful.positive? ? NextStep[:done] : failure_step
|
|
173
166
|
new(ok: true, next_step: step, guidance: step.guidance, payload: batch_result.to_h)
|
|
174
167
|
end
|
|
175
168
|
|
|
176
169
|
def scrape_next_step(empty, botasaurus_configured:)
|
|
177
|
-
return NextStep
|
|
178
|
-
return NextStep
|
|
170
|
+
return NextStep[:done] unless empty
|
|
171
|
+
return NextStep[:read_runtime] unless botasaurus_configured
|
|
179
172
|
|
|
180
|
-
NextStep
|
|
173
|
+
NextStep[:inspect]
|
|
181
174
|
end
|
|
182
175
|
|
|
183
176
|
def scrape_payload(items:, requested_strategy:, channel_title:, admission_drops:)
|
|
@@ -188,33 +181,33 @@ module Html2rss
|
|
|
188
181
|
end
|
|
189
182
|
|
|
190
183
|
def inspect_next_step(report)
|
|
191
|
-
return NextStep
|
|
192
|
-
return NextStep
|
|
184
|
+
return NextStep[:recon] if report.alternate_feeds?
|
|
185
|
+
return NextStep[:capture] if report.articles_count.positive?
|
|
193
186
|
|
|
194
|
-
NextStep
|
|
187
|
+
NextStep[:scrape]
|
|
195
188
|
end
|
|
196
189
|
|
|
197
190
|
def recon_next_step(result)
|
|
198
|
-
return NextStep
|
|
199
|
-
return NextStep
|
|
191
|
+
return NextStep[:done] if result.defer?
|
|
192
|
+
return NextStep[:capture] if result.build?
|
|
200
193
|
|
|
201
|
-
NextStep
|
|
194
|
+
NextStep[:scrape]
|
|
202
195
|
end
|
|
203
196
|
|
|
204
197
|
def capture_next_step(articles_count:, has_selectors:, native_feed: nil)
|
|
205
|
-
return NextStep
|
|
198
|
+
return NextStep[:done] if native_feed
|
|
206
199
|
|
|
207
|
-
articles_count.positive? && has_selectors ? NextStep
|
|
200
|
+
articles_count.positive? && has_selectors ? NextStep[:test] : NextStep[:inspect]
|
|
208
201
|
end
|
|
209
202
|
|
|
210
203
|
def test_next_step(test_result)
|
|
211
|
-
return NextStep
|
|
204
|
+
return NextStep[:apply] if test_result.success
|
|
212
205
|
|
|
213
206
|
kind = test_result.failure_kind
|
|
214
|
-
return NextStep
|
|
215
|
-
return NextStep
|
|
207
|
+
return NextStep[:validate] if kind&.schema?
|
|
208
|
+
return NextStep[:capture] if kind&.execution? || kind&.min_items? || kind&.quality?
|
|
216
209
|
|
|
217
|
-
NextStep
|
|
210
|
+
NextStep[:capture]
|
|
218
211
|
end
|
|
219
212
|
|
|
220
213
|
def test_guidance(test_result, next_step)
|
|
@@ -244,15 +237,15 @@ module Html2rss
|
|
|
244
237
|
|
|
245
238
|
def next_step_for_error(error)
|
|
246
239
|
case error
|
|
247
|
-
when RequestService::BotasaurusConfigurationError then NextStep
|
|
248
|
-
when Contract::UnpublishedRequestError then NextStep
|
|
240
|
+
when RequestService::BotasaurusConfigurationError then NextStep[:read_runtime]
|
|
241
|
+
when Contract::UnpublishedRequestError then NextStep[:validate]
|
|
249
242
|
when ArgumentError then argument_error_next_step(error)
|
|
250
|
-
else NextStep
|
|
243
|
+
else NextStep[:inspect]
|
|
251
244
|
end
|
|
252
245
|
end
|
|
253
246
|
|
|
254
247
|
def argument_error_next_step(error)
|
|
255
|
-
XOR_ERROR.match?(error.message) ? NextStep
|
|
248
|
+
XOR_ERROR.match?(error.message) ? NextStep[:validate] : NextStep[:inspect]
|
|
256
249
|
end
|
|
257
250
|
end
|
|
258
251
|
end
|
|
@@ -4,25 +4,37 @@ module Html2rss
|
|
|
4
4
|
module MCP
|
|
5
5
|
module Server
|
|
6
6
|
##
|
|
7
|
-
# MCP tool
|
|
7
|
+
# MCP tool registry: typed {Tool} values over public html2rss APIs.
|
|
8
8
|
module Tools # rubocop:disable Metrics/ModuleLength -- declarative registry + substantive handlers
|
|
9
|
-
|
|
9
|
+
Tool = Data.define(:name, :title, :description, :input_schema, :annotations, :call)
|
|
10
|
+
|
|
11
|
+
##
|
|
12
|
+
# One published MCP tool. +call+ is a lambda with explicit keyword defaults.
|
|
13
|
+
# +annotations+ is {Contract::ANNOTATIONS_OPEN_WORLD} unless a tool overrides it.
|
|
14
|
+
class Tool # rubocop:disable Lint/EmptyClass -- Data members only; handlers are +call+ lambdas
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
# Published MCP tools consumed by {Server.register_tools}.
|
|
10
18
|
TOOLS = [
|
|
11
|
-
|
|
19
|
+
Tool.new(
|
|
12
20
|
name: 'scrape',
|
|
13
|
-
|
|
21
|
+
title: 'Scrape',
|
|
14
22
|
description: 'One-shot article extraction as JSON Feed items. ' \
|
|
15
23
|
'Use when you need articles now without a saved config. ' \
|
|
16
|
-
'strategy "auto" triggers fallback chain (
|
|
24
|
+
'strategy "auto" triggers fallback chain (default → botasaurus) for JS-rendered sites.',
|
|
17
25
|
input_schema: Contract::SCRAPE_INPUT_SCHEMA,
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
26
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
27
|
+
call: lambda { |url:, strategy: 'auto', limit: 25, items_selector: nil, **|
|
|
28
|
+
scrape_outcome(url:, strategy:, limit:, items_selector:)
|
|
29
|
+
}
|
|
30
|
+
),
|
|
31
|
+
Tool.new(
|
|
21
32
|
name: 'inspect',
|
|
22
|
-
|
|
33
|
+
title: 'Inspect',
|
|
23
34
|
description: 'Diagnostic page analysis (scrapers, SST, segments, final URL, status, ' \
|
|
24
35
|
'rel=alternate feeds). Use recon for BUILD/DEFER/DROP verdict and native_feed preference.',
|
|
25
36
|
input_schema: Contract::INSPECT_INPUT_SCHEMA,
|
|
37
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
26
38
|
call: lambda { |url:, strategy: 'auto', **|
|
|
27
39
|
Outcome.inspect(
|
|
28
40
|
report: PageRecon::Diagnostics.call(
|
|
@@ -30,45 +42,60 @@ module Html2rss
|
|
|
30
42
|
)
|
|
31
43
|
)
|
|
32
44
|
}
|
|
33
|
-
|
|
34
|
-
|
|
45
|
+
),
|
|
46
|
+
Tool.new(
|
|
35
47
|
name: 'recon',
|
|
36
|
-
|
|
48
|
+
title: 'Recon',
|
|
37
49
|
description: 'Curation verdict and native_feed preference for a URL. ' \
|
|
38
50
|
'Use after inspect when alternates warrant deeper recon, or when you need BUILD/DEFER/DROP.',
|
|
39
51
|
input_schema: Contract::RECON_INPUT_SCHEMA,
|
|
52
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
40
53
|
call: lambda { |url:, strategy: 'auto', **|
|
|
41
54
|
Outcome.recon(result: Html2rss.recon(url, strategy: Runtime.coerce_strategy(strategy)))
|
|
42
55
|
}
|
|
43
|
-
|
|
44
|
-
|
|
56
|
+
),
|
|
57
|
+
Tool.new(
|
|
45
58
|
name: 'batch_scrape',
|
|
46
|
-
|
|
47
|
-
batch_method: :batch_scrape,
|
|
48
|
-
limit_default: 10,
|
|
59
|
+
title: 'Batch scrape',
|
|
49
60
|
description: 'Scrape multiple URLs in parallel with per-URL error isolation. ' \
|
|
50
61
|
'Returns structured JSON Feed items and extraction counts.',
|
|
51
|
-
input_schema: Contract::BATCH_SCRAPE_INPUT_SCHEMA
|
|
52
|
-
|
|
53
|
-
|
|
62
|
+
input_schema: Contract::BATCH_SCRAPE_INPUT_SCHEMA,
|
|
63
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
64
|
+
call: lambda { |urls:, strategy: 'auto', concurrency: Batch::DEFAULT_CONCURRENCY, limit: 10, **|
|
|
65
|
+
Outcome.batch_scrape(
|
|
66
|
+
Batch.batch_scrape(urls:, strategy: Runtime.coerce_strategy(strategy), concurrency:, limit:)
|
|
67
|
+
)
|
|
68
|
+
}
|
|
69
|
+
),
|
|
70
|
+
Tool.new(
|
|
54
71
|
name: 'batch_inspect',
|
|
55
|
-
|
|
56
|
-
batch_method: :batch_inspect,
|
|
72
|
+
title: 'Batch inspect',
|
|
57
73
|
description: 'Inspect multiple URLs in parallel with per-URL error isolation. ' \
|
|
58
74
|
'Returns final redirected URLs, status codes, and rel="alternate" feeds.',
|
|
59
|
-
input_schema: Contract::BATCH_INSPECT_INPUT_SCHEMA
|
|
60
|
-
|
|
61
|
-
|
|
75
|
+
input_schema: Contract::BATCH_INSPECT_INPUT_SCHEMA,
|
|
76
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
77
|
+
call: lambda { |urls:, strategy: 'auto', concurrency: Batch::DEFAULT_CONCURRENCY, **|
|
|
78
|
+
Outcome.batch_inspect(
|
|
79
|
+
Batch.batch_inspect(urls:, strategy: Runtime.coerce_strategy(strategy), concurrency:)
|
|
80
|
+
)
|
|
81
|
+
}
|
|
82
|
+
),
|
|
83
|
+
Tool.new(
|
|
62
84
|
name: 'batch_recon',
|
|
63
|
-
|
|
64
|
-
batch_method: :batch_recon,
|
|
85
|
+
title: 'Batch recon',
|
|
65
86
|
description: 'Run recon across multiple URLs in parallel with per-URL error isolation. ' \
|
|
66
87
|
'Returns verdict, native_feed, and surface classification per URL.',
|
|
67
|
-
input_schema: Contract::BATCH_RECON_INPUT_SCHEMA
|
|
68
|
-
|
|
69
|
-
|
|
88
|
+
input_schema: Contract::BATCH_RECON_INPUT_SCHEMA,
|
|
89
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
90
|
+
call: lambda { |urls:, strategy: 'auto', concurrency: Batch::DEFAULT_CONCURRENCY, **|
|
|
91
|
+
Outcome.batch_recon(
|
|
92
|
+
Batch.batch_recon(urls:, strategy: Runtime.coerce_strategy(strategy), concurrency:)
|
|
93
|
+
)
|
|
94
|
+
}
|
|
95
|
+
),
|
|
96
|
+
Tool.new(
|
|
70
97
|
name: 'capture',
|
|
71
|
-
|
|
98
|
+
title: 'Capture',
|
|
72
99
|
description: 'Derive a reusable html2rss feed config from a URL. ' \
|
|
73
100
|
'Use when the goal is a durable YAML (then test → apply). ' \
|
|
74
101
|
'Returns YAML inside payload.yaml (same serializer as CLI capture). ' \
|
|
@@ -76,134 +103,65 @@ module Html2rss
|
|
|
76
103
|
'enhance defaults from admission evidence (false when chrome drops are high). ' \
|
|
77
104
|
'Full schema options live in resource html2rss://schema.',
|
|
78
105
|
input_schema: Contract::CAPTURE_INPUT_SCHEMA,
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
106
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
107
|
+
call: lambda { |url:, strategy: 'auto', items_selector: nil, force: false, topics: nil, title: nil,
|
|
108
|
+
summary: nil, enhance: nil, limit: nil, max_redirects: nil, max_requests: nil, **|
|
|
109
|
+
capture_outcome(url:, strategy:, items_selector:, force:, topics:, title:, summary:,
|
|
110
|
+
enhance:, limit:, max_redirects:, max_requests:)
|
|
111
|
+
}
|
|
112
|
+
),
|
|
113
|
+
Tool.new(
|
|
82
114
|
name: 'validate',
|
|
83
|
-
|
|
115
|
+
title: 'Validate',
|
|
84
116
|
description: 'Validate a feed config hash XOR yaml string against the html2rss JSON schema. ' \
|
|
85
|
-
'Call before test. Failures return isError with payload.
|
|
117
|
+
'Call before test. Failures return isError with payload.issues. ' \
|
|
86
118
|
'Full schema lives in resource html2rss://schema.',
|
|
87
119
|
input_schema: Contract::CONFIG_XOR_SCHEMA,
|
|
88
120
|
annotations: Contract::ANNOTATIONS_VALIDATE,
|
|
89
121
|
call: lambda { |config: nil, yaml: nil, **|
|
|
90
|
-
|
|
91
|
-
Outcome.validate(errors: validation.success? ? nil : validation.errors.to_h)
|
|
122
|
+
Outcome.validate(report: Html2rss::Config.validate(ConfigArgument.parse(config:, yaml:).config))
|
|
92
123
|
}
|
|
93
|
-
|
|
94
|
-
|
|
124
|
+
),
|
|
125
|
+
Tool.new(
|
|
95
126
|
name: 'test',
|
|
96
|
-
|
|
127
|
+
title: 'Test',
|
|
97
128
|
description: 'Validate schema and execute live extraction (asserting >= min_items items). ' \
|
|
98
129
|
'Call after capture or validate; on success next_step is apply. ' \
|
|
99
130
|
'Returns test summary in payload with sample items, timing, failure_kind, ' \
|
|
100
131
|
'and quality_report (warnings for duplicate URLs, junk titles, native feed). ' \
|
|
101
132
|
'Set strict_quality to fail on duplicate URLs, >50% junk titles, or short titles.',
|
|
102
133
|
input_schema: Contract::TEST_INPUT_SCHEMA,
|
|
134
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
103
135
|
call: lambda { |config: nil, yaml: nil, min_items: 1, strict_quality: false,
|
|
104
136
|
compare_enhance: false, **kwargs|
|
|
105
137
|
feed_config = ConfigArgument.parse(config:, yaml:).config
|
|
106
138
|
test_args = { min_items:, strict_quality:, compare_enhance: }
|
|
107
139
|
test_args[:strategy] = Runtime.coerce_strategy(kwargs[:strategy]) if kwargs.key?(:strategy)
|
|
108
|
-
|
|
109
|
-
Outcome.test(test_result)
|
|
140
|
+
Outcome.test(Html2rss.test(feed_config, **test_args))
|
|
110
141
|
}
|
|
111
|
-
|
|
112
|
-
|
|
142
|
+
),
|
|
143
|
+
Tool.new(
|
|
113
144
|
name: 'apply',
|
|
114
|
-
|
|
145
|
+
title: 'Apply',
|
|
115
146
|
description: 'Apply a validated feed config (hash XOR yaml) and return RSS XML in payload.rss. ' \
|
|
116
147
|
'isError when the feed has zero items (ship gate). payload.item_count is RSS item count. ' \
|
|
117
148
|
'Use after test succeeds.',
|
|
118
149
|
input_schema: Contract::APPLY_INPUT_SCHEMA,
|
|
119
|
-
|
|
120
|
-
|
|
150
|
+
annotations: Contract::ANNOTATIONS_OPEN_WORLD,
|
|
151
|
+
call: lambda { |url:, config: nil, yaml: nil, **|
|
|
152
|
+
apply_outcome(url:, config:, yaml:)
|
|
153
|
+
}
|
|
154
|
+
)
|
|
121
155
|
].freeze
|
|
122
156
|
|
|
123
|
-
class << self
|
|
157
|
+
class << self
|
|
124
158
|
##
|
|
125
|
-
#
|
|
126
|
-
#
|
|
127
|
-
# @param
|
|
128
|
-
# @param
|
|
129
|
-
# @
|
|
130
|
-
|
|
131
|
-
TOOLS.each { |entry| register_tool(server, registrar, entry) }
|
|
132
|
-
end
|
|
133
|
-
|
|
134
|
-
private
|
|
135
|
-
|
|
136
|
-
def register_tool(server, registrar, entry)
|
|
137
|
-
case entry[:kind]
|
|
138
|
-
when :url then register_url_tool(server, registrar, entry)
|
|
139
|
-
when :batch then register_batch_tool(server, registrar, entry)
|
|
140
|
-
when :config_xor then register_config_xor_tool(server, registrar, entry)
|
|
141
|
-
when :capture then register_capture_tool(server, registrar, entry)
|
|
142
|
-
else raise ArgumentError, "unknown tool kind: #{entry[:kind].inspect}"
|
|
143
|
-
end
|
|
144
|
-
end
|
|
145
|
-
|
|
146
|
-
def register_url_tool(server, registrar, entry)
|
|
147
|
-
handler = entry[:handler] ? method(entry[:handler]) : entry[:call]
|
|
148
|
-
registrar.call(
|
|
149
|
-
server,
|
|
150
|
-
name: entry[:name],
|
|
151
|
-
description: entry[:description],
|
|
152
|
-
input_schema: entry[:input_schema],
|
|
153
|
-
annotations: entry.fetch(:annotations, Contract::ANNOTATIONS_OPEN_WORLD)
|
|
154
|
-
) do |**kwargs|
|
|
155
|
-
handler.call(**tool_kwargs(kwargs))
|
|
156
|
-
end
|
|
157
|
-
end
|
|
158
|
-
|
|
159
|
-
def register_batch_tool(server, registrar, entry) # rubocop:disable Metrics/MethodLength
|
|
160
|
-
name = entry[:name]
|
|
161
|
-
batch_method = entry[:batch_method]
|
|
162
|
-
limit_default = entry[:limit_default]
|
|
163
|
-
registrar.call(
|
|
164
|
-
server,
|
|
165
|
-
name:,
|
|
166
|
-
description: entry[:description],
|
|
167
|
-
input_schema: entry[:input_schema]
|
|
168
|
-
) do |urls:, strategy: 'auto', **kwargs|
|
|
169
|
-
concurrency = kwargs.fetch(:concurrency, Batch::DEFAULT_CONCURRENCY)
|
|
170
|
-
batch_args = { urls:, strategy: Runtime.coerce_strategy(strategy), concurrency: }
|
|
171
|
-
batch_args[:limit] = kwargs.fetch(:limit, limit_default) unless limit_default.nil?
|
|
172
|
-
Outcome.public_send(name, Batch.public_send(batch_method, **batch_args))
|
|
173
|
-
end
|
|
174
|
-
end
|
|
175
|
-
|
|
176
|
-
def register_config_xor_tool(server, registrar, entry)
|
|
177
|
-
handler = entry[:handler] ? method(entry[:handler]) : entry[:call]
|
|
178
|
-
registrar.call(
|
|
179
|
-
server,
|
|
180
|
-
name: entry[:name],
|
|
181
|
-
description: entry[:description],
|
|
182
|
-
input_schema: entry[:input_schema],
|
|
183
|
-
annotations: entry.fetch(:annotations, Contract::ANNOTATIONS_OPEN_WORLD)
|
|
184
|
-
) do |**kwargs|
|
|
185
|
-
handler.call(**tool_kwargs(kwargs))
|
|
186
|
-
end
|
|
187
|
-
end
|
|
188
|
-
|
|
189
|
-
def register_capture_tool(server, registrar, entry)
|
|
190
|
-
handler = method(entry[:handler])
|
|
191
|
-
registrar.call(
|
|
192
|
-
server,
|
|
193
|
-
name: entry[:name],
|
|
194
|
-
description: entry[:description],
|
|
195
|
-
input_schema: entry[:input_schema]
|
|
196
|
-
) do |**kwargs|
|
|
197
|
-
handler.call(**tool_kwargs(kwargs))
|
|
198
|
-
end
|
|
199
|
-
end
|
|
200
|
-
|
|
201
|
-
def tool_kwargs(inputs)
|
|
202
|
-
args = inputs.dup
|
|
203
|
-
args.delete(:server_context)
|
|
204
|
-
args
|
|
205
|
-
end
|
|
206
|
-
|
|
159
|
+
# @api private
|
|
160
|
+
# @param url [String]
|
|
161
|
+
# @param strategy [String, Symbol]
|
|
162
|
+
# @param limit [Integer]
|
|
163
|
+
# @param items_selector [String, nil]
|
|
164
|
+
# @return [Outcome]
|
|
207
165
|
def scrape_outcome(url:, strategy: 'auto', limit: 25, items_selector: nil)
|
|
208
166
|
wire = Batch.scrape_wire(url:, strategy: Runtime.coerce_strategy(strategy), limit:, items_selector:)
|
|
209
167
|
Outcome.scrape(
|
|
@@ -215,6 +173,20 @@ module Html2rss
|
|
|
215
173
|
)
|
|
216
174
|
end
|
|
217
175
|
|
|
176
|
+
##
|
|
177
|
+
# @api private
|
|
178
|
+
# @param url [String]
|
|
179
|
+
# @param strategy [String, Symbol]
|
|
180
|
+
# @param items_selector [String, nil]
|
|
181
|
+
# @param force [Boolean]
|
|
182
|
+
# @param topics [Array<String>, nil]
|
|
183
|
+
# @param title [String, nil]
|
|
184
|
+
# @param summary [String, nil]
|
|
185
|
+
# @param enhance [Boolean, nil]
|
|
186
|
+
# @param limit [Integer, nil]
|
|
187
|
+
# @param max_redirects [Integer, nil]
|
|
188
|
+
# @param max_requests [Integer, nil]
|
|
189
|
+
# @return [Outcome]
|
|
218
190
|
def capture_outcome(url:, strategy: 'auto', items_selector: nil, force: false, topics: nil, title: nil, # rubocop:disable Metrics/MethodLength, Metrics/ParameterLists
|
|
219
191
|
summary: nil, enhance: nil, limit: nil, max_redirects: nil, max_requests: nil)
|
|
220
192
|
plan = Runtime.coerce_strategy(strategy)
|
|
@@ -245,6 +217,12 @@ module Html2rss
|
|
|
245
217
|
)
|
|
246
218
|
end
|
|
247
219
|
|
|
220
|
+
##
|
|
221
|
+
# @api private
|
|
222
|
+
# @param url [String]
|
|
223
|
+
# @param config [Hash, nil]
|
|
224
|
+
# @param yaml [String, nil]
|
|
225
|
+
# @return [Outcome]
|
|
248
226
|
def apply_outcome(url:, config: nil, yaml: nil)
|
|
249
227
|
feed_config = HashUtil.deep_dup(ConfigArgument.parse(config:, yaml:).config)
|
|
250
228
|
feed_config[:channel] ||= {}
|
|
@@ -253,12 +231,14 @@ module Html2rss
|
|
|
253
231
|
apply_feed_outcome(feed_config, feed_result, outcome)
|
|
254
232
|
end
|
|
255
233
|
|
|
234
|
+
private
|
|
235
|
+
|
|
256
236
|
def apply_feed_outcome(feed_config, feed_result, outcome) # rubocop:disable Metrics/MethodLength -- quality_report + RSS payload
|
|
257
237
|
rss = feed_result.to_rss
|
|
258
238
|
quality_report = Html2rss::Test.quality_report_for(
|
|
259
239
|
rss.items,
|
|
260
240
|
channel_url: feed_config.dig(:channel, :url).to_s,
|
|
261
|
-
|
|
241
|
+
config: Html2rss::Config.from_hash(feed_config),
|
|
262
242
|
feed_result:,
|
|
263
243
|
pipeline_outcome: outcome,
|
|
264
244
|
probe_native_feed: false
|
data/lib/html2rss/mcp/server.rb
CHANGED
|
@@ -10,7 +10,7 @@ module Html2rss
|
|
|
10
10
|
# This module maps MCP kwargs to those APIs, then {Outcome} + {Contract} shape the envelope.
|
|
11
11
|
#
|
|
12
12
|
# Strategy note: MCP +auto+ passes through to FeedPipeline AutoFallback
|
|
13
|
-
# (
|
|
13
|
+
# (default → botasaurus). Concrete strategies are used as-is.
|
|
14
14
|
# Botasaurus requires +BOTASAURUS_SCRAPER_URL+.
|
|
15
15
|
module Server # rubocop:disable Metrics/ModuleLength
|
|
16
16
|
# MCP server display name.
|
|
@@ -20,6 +20,9 @@ module Html2rss
|
|
|
20
20
|
# Loopback bind for HTTP transport (local use only).
|
|
21
21
|
HTTP_BIND_HOST = '127.0.0.1'
|
|
22
22
|
|
|
23
|
+
# Supported transport names.
|
|
24
|
+
TRANSPORTS = Set[:stdio, :http].freeze
|
|
25
|
+
|
|
23
26
|
# Declarative MCP resource registrations consumed by {register_resources}.
|
|
24
27
|
RESOURCES = [
|
|
25
28
|
{
|
|
@@ -29,7 +32,7 @@ module Html2rss
|
|
|
29
32
|
mime_type: 'application/json',
|
|
30
33
|
body: lambda {
|
|
31
34
|
[{ uri: 'html2rss://schema', mimeType: 'application/json',
|
|
32
|
-
text: Html2rss::Config.json_schema_json(pretty: true) }]
|
|
35
|
+
text: Html2rss::Config::Schema.json_schema_json(pretty: true) }]
|
|
33
36
|
}
|
|
34
37
|
},
|
|
35
38
|
{
|
|
@@ -93,7 +96,7 @@ module Html2rss
|
|
|
93
96
|
# @param transport [Symbol] +:stdio+ or +:http+
|
|
94
97
|
# @param port [Integer] port for HTTP transport
|
|
95
98
|
def start(transport: :stdio, port: 8080)
|
|
96
|
-
raise ArgumentError, "Unknown transport: #{transport.inspect}" unless
|
|
99
|
+
raise ArgumentError, "Unknown transport: #{transport.inspect}" unless TRANSPORTS.include?(transport)
|
|
97
100
|
|
|
98
101
|
configure_daemon_logging!
|
|
99
102
|
app = build
|
|
@@ -137,8 +140,12 @@ module Html2rss
|
|
|
137
140
|
|
|
138
141
|
def protocol_configuration
|
|
139
142
|
::MCP::Configuration.new.tap do |config|
|
|
140
|
-
config.exception_reporter =
|
|
141
|
-
|
|
143
|
+
config.exception_reporter = lambda { |error, server_context|
|
|
144
|
+
report_protocol_exception(error, server_context)
|
|
145
|
+
}
|
|
146
|
+
config.around_request = lambda { |data, &block|
|
|
147
|
+
around_protocol_request(data, &block)
|
|
148
|
+
}
|
|
142
149
|
config.validate_tool_call_results = true
|
|
143
150
|
end
|
|
144
151
|
end
|
|
@@ -217,26 +224,27 @@ module Html2rss
|
|
|
217
224
|
tool_error_response(error)
|
|
218
225
|
end
|
|
219
226
|
|
|
227
|
+
def register_tools(server)
|
|
228
|
+
Tools::TOOLS.each { |tool| register_envelope_tool(server, tool) }
|
|
229
|
+
end
|
|
230
|
+
|
|
220
231
|
# rubocop:disable-next Metrics/MethodLength -- listing fields stay together
|
|
221
|
-
def
|
|
222
|
-
|
|
223
|
-
run = method(:handle_tool_call)
|
|
232
|
+
def register_envelope_tool(server, tool)
|
|
233
|
+
invoke = ->(&block) { handle_tool_call(&block) }
|
|
224
234
|
server.define_tool(
|
|
225
|
-
name
|
|
226
|
-
title:
|
|
227
|
-
description
|
|
228
|
-
annotations
|
|
229
|
-
input_schema
|
|
235
|
+
name: tool.name,
|
|
236
|
+
title: tool.title,
|
|
237
|
+
description: tool.description,
|
|
238
|
+
annotations: tool.annotations,
|
|
239
|
+
input_schema: tool.input_schema,
|
|
230
240
|
output_schema: Contract.output_schema
|
|
231
241
|
) do |**kwargs|
|
|
232
|
-
|
|
242
|
+
args = kwargs.dup
|
|
243
|
+
args.delete(:server_context)
|
|
244
|
+
invoke.call { tool.call.call(**args) }
|
|
233
245
|
end
|
|
234
246
|
end
|
|
235
247
|
|
|
236
|
-
def register_tools(server)
|
|
237
|
-
Tools.register_all(server, registrar: method(:define_envelope_tool))
|
|
238
|
-
end
|
|
239
|
-
|
|
240
248
|
def register_resources(server)
|
|
241
249
|
RESOURCES.each do |entry|
|
|
242
250
|
server.define_resource(
|
|
@@ -251,7 +259,7 @@ module Html2rss
|
|
|
251
259
|
end
|
|
252
260
|
|
|
253
261
|
def register_prompts(server) # rubocop:disable Metrics/MethodLength -- prompt argument mapping
|
|
254
|
-
to_result =
|
|
262
|
+
to_result = ->(text) { prompt_result(text) }
|
|
255
263
|
PROMPTS.each do |entry|
|
|
256
264
|
arguments = entry[:arguments].map do |spec|
|
|
257
265
|
::MCP::Prompt::Argument.new(name: spec[:name], description: spec[:description],
|
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
| `alternate_feeds` | `rel=alternate` RSS/Atom links found in HTML |
|
|
14
14
|
| `surface_category` | AutoSource surface class (`high_entropy_surface`, `unsupported_surface`, …) |
|
|
15
15
|
| `articles_count` | Cheap AutoSource extract (limit 10) — diagnostic only, not ship quality |
|
|
16
|
-
| `strategy` | Concrete transport used (`inspect` maps `auto` →
|
|
16
|
+
| `strategy` | Concrete transport used (`inspect` maps `auto` → default (HTTPX)) |
|
|
17
17
|
|
|
18
18
|
CLI text output omits `requested_url`; it prints the requested URL as the card title. The `Final:` line appears **only when** `final_url` differs from `requested_url`:
|
|
19
19
|
|
|
@@ -39,16 +39,16 @@ bin/html2rss inspect https://www.example
|
|
|
39
39
|
# high_entropy_surface, articles (no Final: line — requested equals final)
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
-
Cross-host redirects (e.g. `apex.example` → `www.example`) no longer pin a stale `Host` header from the entry URL. `Config::RequestHeaders` omits `Host` by default;
|
|
42
|
+
Cross-host redirects (e.g. `apex.example` → `www.example`) no longer pin a stale `Host` header from the entry URL. `Config::RequestHeaders` omits `Host` by default; HTTPX sets it per hop from the current request URL.
|
|
43
43
|
|
|
44
|
-
When the first streamed fetch loses the redirect payload (empty body at the final URL), `
|
|
44
|
+
When the first streamed fetch loses the redirect payload (empty body at the final URL), `HttpxStrategy` retries once without streaming so inspect/recon see the same HTML as a direct canonical URL fetch.
|
|
45
45
|
|
|
46
46
|
### What to do
|
|
47
47
|
|
|
48
48
|
1. **Prefer the canonical URL** — if you know the site lives on `www`, pass that URL to inspect, recon, capture, and scrape.
|
|
49
49
|
2. **Read the `Final:` line** — when `final_url` differs from what you typed and status is 4xx, retry inspect on `final_url` before assuming the site is unreachable.
|
|
50
50
|
3. **Do not treat apex 403 as “redirect skipped”** — check JSON output (`--format json`) for `requested_url` vs `final_url` when text output is ambiguous.
|
|
51
|
-
4. **Blocked surfaces** — if the canonical URL still fails, try `strategy: botasaurus` on inspect (MCP) or escalate to recon/capture with browser strategy;
|
|
51
|
+
4. **Blocked surfaces** — if the canonical URL still fails, try `strategy: botasaurus` on inspect (MCP) or escalate to recon/capture with browser strategy; default (HTTPX)-only inspect is intentionally cheap.
|
|
52
52
|
|
|
53
53
|
## inspect ≠ recon
|
|
54
54
|
|
|
@@ -65,5 +65,5 @@ Follow golden-path `next_step` from MCP envelopes; do not call recon when inspec
|
|
|
65
65
|
| ----------------------------------- | -------------------------------------------------------------------- |
|
|
66
66
|
| Diagnostic fetch + assess | `PageRecon::Diagnostics` → `PageRecon.probe` |
|
|
67
67
|
| Surface class + cheap article count | `PageRecon::Assessment` |
|
|
68
|
-
| Redirect follow + terminal retry | `RequestService::
|
|
68
|
+
| Redirect follow + terminal retry | `RequestService::HttpxStrategy` |
|
|
69
69
|
| Outbound header normalization | `Config::RequestHeaders` (no default `Host`; explicit override only) |
|