pikuri-web 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 1ea555be35b6a5f319b0eae6f0957d05014415e4a2987df52e66e33ec2877503
4
+ data.tar.gz: a71c80d161abd74a8b0e6c54b2a5a483a6e7b7206f3df10f1a2b76cb920ce3f6
5
+ SHA512:
6
+ metadata.gz: 9630c95fe7ad66720f7ac8560c4a2ac1380881dcc5cd4f365952f518e65022d164964069497a27592c33307f4b681505cbd8dbb76d8a12201ab3d39da95818ed
7
+ data.tar.gz: 1c9f6b83bc30d76214e5b299bd58659ae891b1303ec798ef1ad998fbcb43c5e5e4c98b8f216e30925cfa563f17f9435fe0855e85c438571ad0aafe7453d13b80
data/README.md ADDED
@@ -0,0 +1,68 @@
1
+ # pikuri-web
2
+
3
+ The tools that reach the internet, for the
4
+ [pikuri](https://github.com/mvysny/pikuri) AI-assistant toolkit:
5
+
6
+ - `Pikuri::Web::Search.build` (`web_search`) — a search across
7
+ DuckDuckGo, plus Brave and Exa when you pass their keys. Built per
8
+ wiring, since the keys come from you, never from the environment.
9
+ - `Pikuri::Web::SCRAPE` (`web_scrape`) — a web page as readable
10
+ Markdown.
11
+ - `Pikuri::Web::FETCH` (`fetch`) — a URL's body verbatim, for JSON,
12
+ CSV, `robots.txt` and source files.
13
+ - `Pikuri::Web::READ_CACHED` (`web_read_cached`) — the page text a
14
+ `page_text: true` search brought back, read by handle, with no
15
+ request of its own.
16
+
17
+ Every one of them puts text written by strangers into the
18
+ conversation, and `web_scrape` / `fetch` will request any URL the
19
+ model types — which is a way to send your data out. That is why they
20
+ live here rather than in pikuri-core: an agent that doesn't need the
21
+ internet doesn't load them, and one that does shows it on the
22
+ trifecta map printed at boot.
23
+
24
+ ## Install
25
+
26
+ ```ruby
27
+ # Gemfile
28
+ gem 'pikuri-web'
29
+ ```
30
+
31
+ ## Usage
32
+
33
+ ```ruby
34
+ require 'pikuri-core'
35
+ require 'pikuri-web'
36
+
37
+ agent = Pikuri::Agent.new(transport: ..., system_prompt: ...) do |c|
38
+ c.add_tool Pikuri::Web::Search.build(brave_key: 'BSA...')
39
+ c.add_tool Pikuri::Web::SCRAPE
40
+ c.add_tool Pikuri::Web::FETCH
41
+ end
42
+ ```
43
+
44
+ For a search-only agent, swap the scraper for the cached reader, so
45
+ the only request that leaves the machine is the search itself:
46
+
47
+ ```ruby
48
+ c.add_tool Pikuri::Web::Search.build(engines: [:exa], exa_key: 'xyz', page_text: true)
49
+ c.add_tool Pikuri::Web::READ_CACHED
50
+ ```
51
+
52
+ For an agent that reads your mail or notes, give it the snippets and
53
+ nothing else, and show yourself every query before it goes out. The
54
+ model may otherwise put "Jan Novák chemo side effects" into a search,
55
+ straight from a mail it just read, and that query then sits in the
56
+ search engine's logs:
57
+
58
+ ```ruby
59
+ c.add_tool Pikuri::Web::Search.build(snippets_only: true,
60
+ confirmer: Pikuri::Workspace::Confirmer::Terminal.new)
61
+ ```
62
+
63
+ ## Further reading
64
+
65
+ - [The first-conversation chapter](../book/chat.md) — the tools in a
66
+ working chat agent, and each search provider's privacy posture.
67
+ - [The sub-agents chapter](../book/subagents.md) — keeping these
68
+ tools off an agent that reads your files, behind a `RESEARCHER`.
@@ -0,0 +1,153 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Pikuri
4
+ module Web
5
+ # Truncation policy and Tool spec for the +fetch+ tool. HTTP work lives in
6
+ # {Web::Scraper.fetch}; this wrapper accepts only textual content-types,
7
+ # applies a character cap, and exposes the result in OpenAI tool-call shape.
8
+ # Sister of {Web::Scrape} but with no extraction pass — bodies are
9
+ # returned verbatim, for raw textual data (JSON APIs, CSV, +robots.txt+,
10
+ # sitemaps, source files) where a rendering pass would corrupt the payload.
11
+ module Fetch
12
+ # @return [Integer] default character cap on the body from {.fetch}.
13
+ # Smaller than {Web::Scrape::DEFAULT_MAX_CHARS} because fetch's
14
+ # content is bimodal — most JSON/XML/CSV is tiny, and large dumps are
15
+ # better re-requested deliberately than padded into every default.
16
+ DEFAULT_MAX_CHARS = 5_000
17
+
18
+ # @return [Integer] hard ceiling on {.fetch}'s +max_chars+; matches
19
+ # {Web::Scrape::MAX_MAX_CHARS}.
20
+ MAX_MAX_CHARS = 100_000
21
+
22
+ # Application content-types that are textual in practice, so safe to
23
+ # return verbatim despite failing the +text/*+ check. Anything outside
24
+ # +text/*+ and this list is refused.
25
+ # @return [Array<String>]
26
+ TEXTUAL_APPLICATION_TYPES = %w[
27
+ application/json
28
+ application/xml
29
+ application/javascript
30
+ application/xhtml+xml
31
+ application/rss+xml
32
+ application/atom+xml
33
+ ].freeze
34
+
35
+ # On-disk cache for {.fetch}, in its own subdir so a +fetch+ and a
36
+ # +web_scrape+ on the same URL don't collide (raw body vs extracted
37
+ # Markdown). A method so specs can swap {UrlCache::NULL}.
38
+ #
39
+ # @return [UrlCache::OnDisk, #fetch]
40
+ CACHE = UrlCache::OnDisk.new(ttl: UrlCache::DEFAULT_TTL, dir: "#{UrlCache::ROOT_DIR}/fetch")
41
+ # Accessor for {CACHE}; specs swap in {UrlCache::NULL} or an isolated cache.
42
+ #
43
+ # @return [UrlCache::OnDisk, #fetch]
44
+ def self.cache
45
+ CACHE
46
+ end
47
+
48
+ # Download +url+ via {Web::Scraper.fetch} and return the body verbatim,
49
+ # if the content-type is textual (any +text/*+ plus
50
+ # {TEXTUAL_APPLICATION_TYPES}); anything else (PDF, image, binary) yields
51
+ # an +"Error: ..."+ string. Cached on disk keyed by URL; +max_chars+ is
52
+ # *not* in the key (values share one entry, truncation runs after the
53
+ # lookup). Cached only on success — {Scraper::FetchError} is caught
54
+ # outside +cache.fetch+, so failures are never persisted; other
55
+ # exceptions bubble up.
56
+ #
57
+ # @param url [String] absolute HTTP(S) URL
58
+ # @param max_chars [Integer] body cap, clamped to +[1, {MAX_MAX_CHARS}]+;
59
+ # default {DEFAULT_MAX_CHARS}. Over the cap, output is cut with a marker.
60
+ # @return [Tool::Result] the body, possibly truncated (+"Fetched 4.2 kB"+),
61
+ # or {Pikuri::Tool::Result.error} on a recoverable failure
62
+ def self.fetch(url, max_chars: DEFAULT_MAX_CHARS)
63
+ max_chars = max_chars.clamp(1, MAX_MAX_CHARS)
64
+ body = cache.fetch(url) { download(url) }
65
+ Tool::Result.new(content: truncate(body, max_chars),
66
+ summary: Scraper.transfer_summary(verb: 'Fetched', full: body, max_chars: max_chars))
67
+ rescue Scraper::FetchError => e
68
+ Tool::Result.error(e.message)
69
+ end
70
+
71
+ # GET +url+ and verify the content-type is textual. Always hits the
72
+ # network (caller caches/truncates).
73
+ #
74
+ # @param url [String]
75
+ # @return [String] response body
76
+ # @raise [Scraper::FetchError] on HTTP non-2xx, network failure, redirect
77
+ # exhaustion, missing +Location+, or a non-textual content-type
78
+ def self.download(url)
79
+ fetched = Scraper.fetch(url)
80
+ return fetched.body if textual?(fetched.content_type)
81
+
82
+ raise Scraper::FetchError,
83
+ "refused to fetch #{url}: content-type #{fetched.content_type.inspect} " \
84
+ 'is not textual (use web_scrape for rendered pages)'
85
+ end
86
+
87
+ # @param content_type [String] normalized content-type from {Scraper.fetch}
88
+ # @return [Boolean] true when +text/*+ or in {TEXTUAL_APPLICATION_TYPES}
89
+ def self.textual?(content_type)
90
+ content_type.start_with?('text/') ||
91
+ TEXTUAL_APPLICATION_TYPES.include?(content_type)
92
+ end
93
+
94
+ # Cut +body+ to at most +max_chars+, appending a marker with the original
95
+ # length when truncation happens. Same shape as {Web::Scrape.truncate}.
96
+ #
97
+ # @param body [String] full response body
98
+ # @param max_chars [Integer] cap; assumed already clamped
99
+ # @return [String]
100
+ def self.truncate(body, max_chars)
101
+ return body if body.length <= max_chars
102
+
103
+ "#{body[0, max_chars]}\n\n" \
104
+ "... [truncated at #{max_chars} of #{body.length} chars; " \
105
+ 'call again with a larger `max_chars` to see more]'
106
+ end
107
+ end
108
+
109
+ # Verbatim URL download tool — a thin wrapper over {Web::Fetch.fetch} in
110
+ # OpenAI tool-call shape. Use for raw textual payloads (JSON, CSV,
111
+ # +robots.txt+, source files); use {Web::SCRAPE} for rendered pages.
112
+ #
113
+ # No confirmer may ever auto-approve a URL that "looks clean": a secret
114
+ # leaves through a bare path (+paste.example/SECRET+), a subdomain (the DNS
115
+ # lookup alone reaches the attacker's nameserver) or a spotless paste-site
116
+ # URL the attacker reads back. Even a confirmer showing the whole URL makes
117
+ # this +human_reviewed+, never safe: the model still picks the host.
118
+ #
119
+ # Sharing: +P_shared_benign+ — no per-agent state, but {Fetch::CACHE} is
120
+ # process-wide (and on disk, so cross-process too): agent B's +fetch+ can
121
+ # be answered from agent A's download within the TTL. See
122
+ # {Pikuri::Web::UrlCache::OnDisk} for the race and why it's harmless.
123
+ #
124
+ # @return [Tool]
125
+ FETCH = Tool.new(
126
+ name: 'fetch',
127
+ description: <<~DESC,
128
+ Downloads the given URL and returns its body verbatim.
129
+
130
+ Usage:
131
+ - Use for raw textual payloads: JSON APIs, CSV files, robots.txt, sitemaps, source files — anywhere a rendering pass would corrupt the data.
132
+ - For rendered HTML pages, use web_scrape — it extracts readable content; fetch returns the raw HTML bytes unchanged.
133
+ - Accepts text/* and common textual application/* types (JSON, XML, JS, XHTML, RSS, Atom). Refuses PDFs, images, and other binaries.
134
+ DESC
135
+ parameters: Tool::Parameters.build { |p|
136
+ p.required_string :url,
137
+ 'Absolute URL to download, including the scheme, ' \
138
+ 'e.g. "https://example.com/data.json".'
139
+ p.optional_integer :max_chars,
140
+ 'Maximum number of characters of the body to ' \
141
+ "return. Defaults to #{Fetch::DEFAULT_MAX_CHARS}; hard-capped at " \
142
+ "#{Fetch::MAX_MAX_CHARS}. When the body is longer than this, " \
143
+ 'output is cut and a marker reports the full ' \
144
+ 'length.'
145
+ },
146
+ execute: ->(url:, max_chars: Fetch::DEFAULT_MAX_CHARS) {
147
+ Fetch.fetch(url, max_chars: max_chars)
148
+ },
149
+ # As {SCRAPE}: attacker-choosable host, attacker-authored response.
150
+ trifecta_legs: Pikuri::Tool::TrifectaLegs::ASSUMED
151
+ )
152
+ end
153
+ end
@@ -0,0 +1,96 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'time'
4
+
5
+ module Pikuri
6
+ module Web
7
+ # Reading policy for the +web_read_cached+ tool: a search result's page
8
+ # text, from the {Search::PageStore} a +page_text+ {Search} filled —
9
+ # never from the page's origin.
10
+ #
11
+ # ReadCached.read('a3f9c21e').content
12
+ # # => "URL: https://docs.ruby-lang.org/en/3.4/Data.html\nTitle: Data\n
13
+ # # Indexed: 2026-11-10T05:22:14Z\n\nclass Data …"
14
+ #
15
+ # Takes a handle, never a URL: a URL would mean a fetch, and the page a
16
+ # model picks is a channel (+D_no_result_scraper+) — let alone a URL it
17
+ # writes. For the same reason links inside a page can't be followed. The
18
+ # text is the engine's snapshot — Brave's in excerpts, possibly older than
19
+ # the live page — so the read says when the engine fetched it, when known.
20
+ #
21
+ # Named +web_+ for the family, +read+ for pikuri's verb for "give me the
22
+ # content", +cached+ because it is no live fetch (+web_cache+ reads as an
23
+ # order to cache something).
24
+ module ReadCached
25
+ # Read the page behind +handle+, cut to +max_chars+ with {Scrape}'s
26
+ # marker.
27
+ #
28
+ # @param handle [String] as printed in the search results, e.g. +"a3f9c21e"+
29
+ # @param max_chars [Integer] text cap, clamped to
30
+ # +[1, {Scrape::MAX_MAX_CHARS}]+; default {Scrape::DEFAULT_MAX_CHARS}
31
+ # @param store [Search::PageStore]
32
+ # @return [Tool::Result] the URL, title, index time and text (+"Read
33
+ # 12.4 kB"+), or {Tool::Result.error} for an unknown or evicted handle
34
+ def self.read(handle, max_chars: Scrape::DEFAULT_MAX_CHARS, store: Search::PageStore.default)
35
+ page = store.read(handle)
36
+ if page.nil?
37
+ return Tool::Result.error(
38
+ "No stored page for handle #{handle.to_s.strip.inspect}. " \
39
+ 'Run web_search again and use a handle from its results.'
40
+ )
41
+ end
42
+
43
+ max_chars = max_chars.clamp(1, Scrape::MAX_MAX_CHARS)
44
+ header = "URL: #{page.url}\nTitle: #{page.title}\n"
45
+ header += "Indexed: #{page.indexed_at.utc.iso8601}\n" if page.indexed_at
46
+ Tool::Result.new(content: "#{header}\n#{Scrape.truncate(page.text, max_chars)}",
47
+ summary: Scraper.transfer_summary(verb: 'Read', full: page.text, max_chars: max_chars))
48
+ end
49
+ end
50
+
51
+ # A web_search result's stored page text — a thin wrapper over
52
+ # {Web::ReadCached.read} in OpenAI tool-call shape. Pairs with a
53
+ # +page_text+ {Search.build}; wired without one, every read answers
54
+ # "search again".
55
+ #
56
+ # Sends nothing: no network request at all, so the bytes it returns came
57
+ # in with the search, and what left the machine was that search's query —
58
+ # see each engine's +Privacy posture+ ({Search::Brave}, {Search::Exa}).
59
+ #
60
+ # Sharing: +P_shared_locked+ — no state of its own; it reads the
61
+ # process-wide {Search::PageStore}, whose +Mutex+ guards it.
62
+ #
63
+ # @return [Tool]
64
+ READ_CACHED = Tool.new(
65
+ name: 'web_read_cached',
66
+ description: <<~DESC,
67
+ Returns the full text of a web_search result's page.
68
+
69
+ Usage:
70
+ - Takes a handle from web_search's results — the hex code in brackets before a result's title, e.g. [a3f9c21e] — never a URL. A result without a handle can't be read.
71
+ - When a handle is unknown, run web_search again and use a handle from its results.
72
+ - The text may come in excerpts and be older than the live page; the Indexed line, when present, says when it was captured.
73
+ - Links inside the page can't be followed: only pages web_search returned are readable.
74
+ DESC
75
+ parameters: Tool::Parameters.build { |p|
76
+ p.required_string :handle,
77
+ 'Handle of a web_search result, as printed in brackets ' \
78
+ 'before its title, e.g. "a3f9c21e".'
79
+ p.optional_integer :max_chars,
80
+ 'Maximum number of characters of page text to ' \
81
+ "return. Defaults to #{Scrape::DEFAULT_MAX_CHARS}; hard-capped at " \
82
+ "#{Scrape::MAX_MAX_CHARS}. When the page is longer than this, " \
83
+ 'output is cut and a marker reports the full ' \
84
+ 'length.'
85
+ },
86
+ execute: ->(handle:, max_chars: Scrape::DEFAULT_MAX_CHARS) {
87
+ ReadCached.read(handle, max_chars: max_chars)
88
+ },
89
+ # No request, so no egress leg; the text is whatever the page's author
90
+ # wrote, verbatim into context, hence +:hard+.
91
+ trifecta_legs: Pikuri::Tool::TrifectaLegs.new(
92
+ private: false, untrusted: :hard, egress_payload_review: :no_egress
93
+ )
94
+ )
95
+ end
96
+ end
@@ -0,0 +1,107 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Pikuri
4
+ module Web
5
+ # Truncation policy and Tool spec for the +web_scrape+ tool. Scraping lives
6
+ # in {Web::Scraper}; this wrapper applies a character cap and exposes the
7
+ # result in OpenAI tool-call shape.
8
+ module Scrape
9
+ # @return [Integer] default character cap on the Markdown from {.visit}.
10
+ # Sized to cover most post-readability article bodies in full on the
11
+ # first call, so the LLM doesn't re-request and duplicate the prefix in
12
+ # its context. Genuinely long pages still get cut, and the marker invites
13
+ # a deliberate larger call.
14
+ DEFAULT_MAX_CHARS = 20_000
15
+
16
+ # @return [Integer] hard ceiling on {.visit}'s +max_chars+; requests above
17
+ # it are clamped silently so the LLM can't dump an arbitrarily large page.
18
+ MAX_MAX_CHARS = 100_000
19
+
20
+ # On-disk cache for {.visit}. A method so specs can swap {UrlCache::NULL}.
21
+ #
22
+ # @return [UrlCache::OnDisk, #fetch]
23
+ CACHE = UrlCache::OnDisk.new(ttl: UrlCache::DEFAULT_TTL, dir: "#{UrlCache::ROOT_DIR}/web_scrape")
24
+ # Accessor for {CACHE}; specs swap in {UrlCache::NULL} or an isolated cache.
25
+ #
26
+ # @return [UrlCache::OnDisk, #fetch]
27
+ def self.cache
28
+ CACHE
29
+ end
30
+
31
+ # Fetch +url+ via {Web::Scraper} and truncate the rendered Markdown to
32
+ # +max_chars+. The full Markdown is cached on disk keyed by URL (repeat
33
+ # visits skip the network and extraction); +max_chars+ is *not* in the key
34
+ # (values share one entry, truncation runs after the lookup).
35
+ # {Scraper::FetchError} is caught outside +cache.fetch+ and returned as
36
+ # +"Error: ..."+ (so the LLM can retry a different URL) — never persisted;
37
+ # other exceptions bubble up.
38
+ #
39
+ # @param url [String] absolute HTTP(S) URL
40
+ # @param max_chars [Integer] Markdown cap, clamped to +[1, {MAX_MAX_CHARS}]+;
41
+ # default {DEFAULT_MAX_CHARS}. Over the cap, output is cut with a marker.
42
+ # @return [Tool::Result] the Markdown, possibly truncated (+"Scraped 12.4
43
+ # kB"+), or {Pikuri::Tool::Result.error} on a recoverable fetch failure
44
+ def self.visit(url, max_chars: DEFAULT_MAX_CHARS)
45
+ max_chars = max_chars.clamp(1, MAX_MAX_CHARS)
46
+ markdown = cache.fetch(url) { Scraper.visit(url) }
47
+ Tool::Result.new(content: truncate(markdown, max_chars),
48
+ summary: Scraper.transfer_summary(verb: 'Scraped', full: markdown,
49
+ max_chars: max_chars))
50
+ rescue Scraper::FetchError => e
51
+ Tool::Result.error(e.message)
52
+ end
53
+
54
+ # Cut +markdown+ to at most +max_chars+, appending a marker with the
55
+ # original length when truncation happens.
56
+ #
57
+ # @param markdown [String] full Markdown text
58
+ # @param max_chars [Integer] cap; assumed already clamped
59
+ # @return [String]
60
+ def self.truncate(markdown, max_chars)
61
+ return markdown if markdown.length <= max_chars
62
+
63
+ "#{markdown[0, max_chars]}\n\n" \
64
+ "... [truncated at #{max_chars} of #{markdown.length} chars; " \
65
+ 'call again with a larger `max_chars` to see more]'
66
+ end
67
+ end
68
+
69
+ # Webpage download + Markdown conversion tool — a thin wrapper over
70
+ # {Web::Scrape.visit} in OpenAI tool-call shape.
71
+ #
72
+ # Sharing: +P_shared_benign+ — as {Web::FETCH}: no per-agent state, but
73
+ # {Scrape::CACHE} is process-wide and on disk, so one agent's scrape
74
+ # answers another's within the TTL. See {Pikuri::Web::UrlCache::OnDisk}.
75
+ #
76
+ # @return [Tool]
77
+ SCRAPE = Tool.new(
78
+ name: 'web_scrape',
79
+ description: <<~DESC,
80
+ Scrapes the rendered webpage or text file at the given URL and returns its main content as Markdown.
81
+
82
+ Usage:
83
+ - Use for HTML pages where you want readable content — readability extraction strips nav, sidebars, and boilerplate.
84
+ - For raw textual payloads (JSON, CSV, robots.txt, source files), use fetch instead — it returns bytes verbatim, while web_scrape would corrupt them with a Markdown pass.
85
+ - A Single Page App may return very little or no content. Do NOT retry with a larger max_chars; try a different URL instead.
86
+ DESC
87
+ parameters: Tool::Parameters.build { |p|
88
+ p.required_string :url,
89
+ 'Absolute URL of the webpage to scrape, including ' \
90
+ 'the scheme, e.g. "https://example.com/article".'
91
+ p.optional_integer :max_chars,
92
+ 'Maximum number of characters of Markdown to ' \
93
+ "return. Defaults to #{Scrape::DEFAULT_MAX_CHARS}; hard-capped at " \
94
+ "#{Scrape::MAX_MAX_CHARS}. When the page is longer than this, " \
95
+ 'output is cut and a marker reports the full ' \
96
+ 'length.'
97
+ },
98
+ execute: ->(url:, max_chars: Scrape::DEFAULT_MAX_CHARS) {
99
+ Scrape.visit(url, max_chars: max_chars)
100
+ },
101
+ # Exactly {TrifectaLegs::ASSUMED}, and the archetype it was named for: the
102
+ # page is authored by whoever runs the site, and the model picks the URL,
103
+ # so the request is itself an outbound channel.
104
+ trifecta_legs: Pikuri::Tool::TrifectaLegs::ASSUMED
105
+ )
106
+ end
107
+ end
@@ -0,0 +1,188 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'faraday'
4
+ require 'stringio'
5
+ require 'uri'
6
+
7
+ module Pikuri
8
+ module Web
9
+ # HTTP side of the web tools ({Web::SCRAPE} and {Web::FETCH}): GET the
10
+ # URL with a real-browser User-Agent, follow redirects, and hand the body to
11
+ # {Pikuri::Extractor.extract} with the response +Content-Type+ as the hint.
12
+ # HTML/XHTML renders via {Pikuri::Extractors::HTML}, other +text/*+ passes through
13
+ # verbatim, plug-in extractors extend the set (with pikuri-pdf,
14
+ # +application/pdf+ extracts — by header or +%PDF-+ magic, so a lying header
15
+ # still works); remaining types raise {FetchError} so the LLM observes the
16
+ # failure rather than an empty rendering.
17
+ #
18
+ # Split into a thin HTTP fetch ({.fetch}) and the extraction wrapper
19
+ # ({.visit}) so tests drive each in isolation and {Web::Fetch} can reuse the
20
+ # HTTP half. Nothing here knows about the LLM — the wrapping tools own
21
+ # caching, truncation, and turning the result into an observation.
22
+ module Scraper
23
+ # Raised when a URL can't be rendered to Markdown — HTTP non-2xx, network
24
+ # failure, redirect loop, missing +Location+, unsupported content-type, or
25
+ # a parse failure. The web tools rescue this into an +"Error: ..."+
26
+ # observation; anything else bubbles up so genuine bugs stay visible.
27
+ class FetchError < StandardError; end
28
+
29
+ # @return [String] User-Agent sent with each request: Firefox ESR on
30
+ # Linux. Sites reject a bot-looking UA and some a stale one (technical.city
31
+ # drops +Chrome/120+), so bump it to each new ESR, roughly yearly. ESR,
32
+ # not rapid release: a rapid-release version goes stale in weeks, on a
33
+ # cadence Mozilla keeps changing.
34
+ USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0'
35
+ # @return [String] +Accept+ header, so content-negotiating servers hand
36
+ # back something usable: HTML first, then +application/pdf+, then +text/*+.
37
+ ACCEPT = 'text/html,application/xhtml+xml,application/pdf,text/*;q=0.8'
38
+ # @return [Integer] maximum HTTP redirects to follow before giving up.
39
+ MAX_REDIRECTS = 5
40
+ # @return [Integer] connect timeout (s) for the Faraday request.
41
+ OPEN_TIMEOUT = 10
42
+ # @return [Integer] read timeout (s) for the Faraday request.
43
+ READ_TIMEOUT = 20
44
+
45
+ # @return [Integer] max chars of an error response body in a {FetchError}
46
+ # message — enough to show what came back (a Cloudflare/WAF page) without
47
+ # flooding the observation.
48
+ ERROR_BODY_EXCERPT = 200
49
+
50
+ # Result of a successful {Scraper.fetch}: the response body, the
51
+ # normalized content-type (lower-cased, with any +; charset=...+
52
+ # parameters stripped), and the final URL after redirects.
53
+ Fetched = Data.define(:body, :content_type, :url)
54
+
55
+ # Fetch +url+ and render its main content as Markdown. No caching — every
56
+ # call hits the network (wrap it to memoize; {Web::Scrape.visit}
57
+ # does). The output is +strip+'d (leading/trailing blank lines are common
58
+ # from PDF page-feeds and trailing newlines) but interior whitespace is
59
+ # preserved (paragraph breaks and code indentation are load-bearing).
60
+ #
61
+ # @param url [String] absolute HTTP(S) URL
62
+ # @return [String] Markdown, whitespace-trimmed, otherwise uncapped —
63
+ # caller size-limits before feeding the LLM
64
+ # @raise [FetchError] on HTTP non-2xx, network failure, redirect loop, a
65
+ # 3xx without +Location+, an unrecognized response, or extraction failure
66
+ def self.visit(url)
67
+ extract(fetch(url)).strip
68
+ end
69
+
70
+ # Render a {Fetched} response as Markdown via {Pikuri::Extractor.extract},
71
+ # re-raising both extraction failure modes as {FetchError}. The
72
+ # content-type is passed verbatim (including +""+ for a missing header,
73
+ # which matches no text arm — a body without transport metadata is
74
+ # refused, not sniffed; only a strong magic sniff like +%PDF-+ overrides).
75
+ #
76
+ # @param fetched [Fetched]
77
+ # @return [String] Markdown from the matched extractor
78
+ # @raise [FetchError] when no extractor matches, or extraction fails
79
+ def self.extract(fetched)
80
+ Pikuri::Extractor.extract(StringIO.new(fetched.body), content_type: fetched.content_type)
81
+ rescue Pikuri::Extractor::Unsupported
82
+ raise FetchError, "unsupported content-type #{fetched.content_type.inspect} for #{fetched.url}"
83
+ rescue Pikuri::Extractor::Error => e
84
+ raise FetchError, e.message
85
+ end
86
+
87
+ # Download +url+, manually following up to {MAX_REDIRECTS} redirects. All
88
+ # recoverable failures (HTTP 4xx/5xx, +Faraday::Error+, exhausted redirect
89
+ # budget, 3xx without +Location+) surface as {FetchError}. Error bodies
90
+ # are trimmed to {ERROR_BODY_EXCERPT} so a challenge page doesn't dump
91
+ # kilobytes into the observation.
92
+ #
93
+ # @param url [String] absolute HTTP(S) URL
94
+ # @param limit [Integer] redirects remaining; recurses with +limit - 1+ on 3xx
95
+ # @return [Fetched] body, normalized content-type, final URL
96
+ # @raise [FetchError] on non-2xx/3xx, network error, redirect-loop
97
+ # exhaustion, or 3xx without +Location+
98
+ def self.fetch(url, limit: MAX_REDIRECTS)
99
+ raise FetchError, "too many redirects fetching #{url}" if limit.zero?
100
+
101
+ response = begin
102
+ browser_connection(url).get(url) { |req| req.headers['Accept'] = ACCEPT }
103
+ rescue Faraday::Error => e
104
+ raise FetchError, "#{e.class.name.split('::').last} fetching #{url}: #{e.message}"
105
+ end
106
+
107
+ case response.status
108
+ when 200..299
109
+ Fetched.new(body: response.body, content_type: normalize_content_type(response.headers['content-type']), url: url)
110
+ when 300..399
111
+ location = response.headers['location']
112
+ raise FetchError, "HTTP #{response.status} from #{url} with no Location header" if location.nil? || location.empty?
113
+
114
+ fetch(URI.join(url, location).to_s, limit: limit - 1)
115
+ else
116
+ raise FetchError, "HTTP #{response.status} fetching #{url}: #{excerpt(response.body)}"
117
+ end
118
+ end
119
+
120
+ # A Faraday connection that imitates a browser — Firefox — to +url+'s host:
121
+ # +Host+ first, then {USER_AGENT}, with the {OPEN_TIMEOUT} / {READ_TIMEOUT}
122
+ # budget. For a page meant for people in a browser, which a site defends
123
+ # against bots (web pages, DuckDuckGo's HTML results):
124
+ #
125
+ # Scraper.browser_connection(ENDPOINT).get(ENDPOINT, { q: query })
126
+ #
127
+ # Not for an API meant for programs (Brave's, Exa's): call plain Faraday
128
+ # there, as the client pikuri is — imitating a browser buys nothing.
129
+ # Headers set on a request land after these two, keeping the browser order.
130
+ #
131
+ # @param url [String] absolute HTTP(S) URL; its host and port become +Host+
132
+ # @return [Faraday::Connection] good for that one host only
133
+ def self.browser_connection(url)
134
+ uri = URI(url)
135
+ # Host first, as browsers send it: Net::HTTP would append it last, and a
136
+ # browser UA with a trailing Host draws Cloudflare's 403 challenge.
137
+ headers = {
138
+ 'Host' => uri.port == uri.default_port ? uri.host : "#{uri.host}:#{uri.port}",
139
+ 'User-Agent' => USER_AGENT
140
+ }
141
+ Faraday.new(headers: headers, request: { open_timeout: OPEN_TIMEOUT, timeout: READ_TIMEOUT })
142
+ end
143
+
144
+ # Display line for a downloaded body — what the model got, plus the full
145
+ # size when the cap cut it:
146
+ #
147
+ # Scraper.transfer_summary(verb: 'Fetched', full: body, max_chars: 5_000)
148
+ # # => "Fetched 5.0 kB of 41.2 kB"
149
+ #
150
+ # No content-type in it: both wrappers cache the *body*, so a cache hit
151
+ # has no header to report, and a line that came and went with the cache
152
+ # would be worse than one that never names the type.
153
+ #
154
+ # @param verb [String] +"Fetched"+ or +"Scraped"+
155
+ # @param full [String] the whole body, before truncation
156
+ # @param max_chars [Integer] the cap applied to it
157
+ # @return [String]
158
+ def self.transfer_summary(verb:, full:, max_chars:)
159
+ return "#{verb} #{Pikuri::Tool.format_bytes(full.bytesize)}" if full.length <= max_chars
160
+
161
+ "#{verb} #{Pikuri::Tool.format_bytes(full[0, max_chars].bytesize)} " \
162
+ "of #{Pikuri::Tool.format_bytes(full.bytesize)}"
163
+ end
164
+
165
+ # Lower-case +raw+ and strip any +; charset=...+ parameters so the
166
+ # extractors can match on a canonical token.
167
+ #
168
+ # @param raw [String, nil] raw +Content-Type+ header value
169
+ # @return [String] normalized content-type, or +""+ when the
170
+ # header was missing
171
+ def self.normalize_content_type(raw)
172
+ raw.to_s.split(';').first.to_s.strip.downcase
173
+ end
174
+ private_class_method :normalize_content_type
175
+
176
+ # Whitespace-collapse +body+ and clip to {ERROR_BODY_EXCERPT}, keeping the
177
+ # {FetchError} message a single readable line.
178
+ #
179
+ # @param body [String, nil]
180
+ # @return [String]
181
+ def self.excerpt(body)
182
+ text = body.to_s.gsub(/\s+/, ' ').strip
183
+ text.length > ERROR_BODY_EXCERPT ? "#{text[0, ERROR_BODY_EXCERPT]}..." : text
184
+ end
185
+ private_class_method :excerpt
186
+ end
187
+ end
188
+ end