pikuri-web 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/README.md +68 -0
- data/lib/pikuri/web/fetch.rb +153 -0
- data/lib/pikuri/web/read_cached.rb +96 -0
- data/lib/pikuri/web/scrape.rb +107 -0
- data/lib/pikuri/web/scraper.rb +188 -0
- data/lib/pikuri/web/search/brave.rb +298 -0
- data/lib/pikuri/web/search/duckduckgo.rb +181 -0
- data/lib/pikuri/web/search/engines.rb +430 -0
- data/lib/pikuri/web/search/exa.rb +243 -0
- data/lib/pikuri/web/search/page_store.rb +142 -0
- data/lib/pikuri/web/search/rate_limiter.rb +114 -0
- data/lib/pikuri/web/search/result.rb +42 -0
- data/lib/pikuri/web/search.rb +229 -0
- data/lib/pikuri/web/url_cache/in_memory.rb +96 -0
- data/lib/pikuri/web/url_cache/on_disk.rb +80 -0
- data/lib/pikuri/web/url_cache.rb +63 -0
- data/lib/pikuri-web.rb +36 -0
- metadata +138 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 1ea555be35b6a5f319b0eae6f0957d05014415e4a2987df52e66e33ec2877503
|
|
4
|
+
data.tar.gz: a71c80d161abd74a8b0e6c54b2a5a483a6e7b7206f3df10f1a2b76cb920ce3f6
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 9630c95fe7ad66720f7ac8560c4a2ac1380881dcc5cd4f365952f518e65022d164964069497a27592c33307f4b681505cbd8dbb76d8a12201ab3d39da95818ed
|
|
7
|
+
data.tar.gz: 1c9f6b83bc30d76214e5b299bd58659ae891b1303ec798ef1ad998fbcb43c5e5e4c98b8f216e30925cfa563f17f9435fe0855e85c438571ad0aafe7453d13b80
|
data/README.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# pikuri-web
|
|
2
|
+
|
|
3
|
+
The tools that reach the internet, for the
|
|
4
|
+
[pikuri](https://github.com/mvysny/pikuri) AI-assistant toolkit:
|
|
5
|
+
|
|
6
|
+
- `Pikuri::Web::Search.build` (`web_search`) — a search across
|
|
7
|
+
DuckDuckGo, plus Brave and Exa when you pass their keys. Built per
|
|
8
|
+
wiring, since the keys come from you, never from the environment.
|
|
9
|
+
- `Pikuri::Web::SCRAPE` (`web_scrape`) — a web page as readable
|
|
10
|
+
Markdown.
|
|
11
|
+
- `Pikuri::Web::FETCH` (`fetch`) — a URL's body verbatim, for JSON,
|
|
12
|
+
CSV, `robots.txt` and source files.
|
|
13
|
+
- `Pikuri::Web::READ_CACHED` (`web_read_cached`) — the page text a
|
|
14
|
+
`page_text: true` search brought back, read by handle, with no
|
|
15
|
+
request of its own.
|
|
16
|
+
|
|
17
|
+
Every one of them puts text written by strangers into the
|
|
18
|
+
conversation, and `web_scrape` / `fetch` will request any URL the
|
|
19
|
+
model types — which is a way to send your data out. That is why they
|
|
20
|
+
live here rather than in pikuri-core: an agent that doesn't need the
|
|
21
|
+
internet doesn't load them, and one that does shows it on the
|
|
22
|
+
trifecta map printed at boot.
|
|
23
|
+
|
|
24
|
+
## Install
|
|
25
|
+
|
|
26
|
+
```ruby
|
|
27
|
+
# Gemfile
|
|
28
|
+
gem 'pikuri-web'
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Usage
|
|
32
|
+
|
|
33
|
+
```ruby
|
|
34
|
+
require 'pikuri-core'
|
|
35
|
+
require 'pikuri-web'
|
|
36
|
+
|
|
37
|
+
agent = Pikuri::Agent.new(transport: ..., system_prompt: ...) do |c|
|
|
38
|
+
c.add_tool Pikuri::Web::Search.build(brave_key: 'BSA...')
|
|
39
|
+
c.add_tool Pikuri::Web::SCRAPE
|
|
40
|
+
c.add_tool Pikuri::Web::FETCH
|
|
41
|
+
end
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
For a search-only agent, swap the scraper for the cached reader, so
|
|
45
|
+
the only request that leaves the machine is the search itself:
|
|
46
|
+
|
|
47
|
+
```ruby
|
|
48
|
+
c.add_tool Pikuri::Web::Search.build(engines: [:exa], exa_key: 'xyz', page_text: true)
|
|
49
|
+
c.add_tool Pikuri::Web::READ_CACHED
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
For an agent that reads your mail or notes, give it the snippets and
|
|
53
|
+
nothing else, and show yourself every query before it goes out. The
|
|
54
|
+
model may otherwise put "Jan Novák chemo side effects" into a search,
|
|
55
|
+
straight from a mail it just read, and that query then sits in the
|
|
56
|
+
search engine's logs:
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
c.add_tool Pikuri::Web::Search.build(snippets_only: true,
|
|
60
|
+
confirmer: Pikuri::Workspace::Confirmer::Terminal.new)
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## Further reading
|
|
64
|
+
|
|
65
|
+
- [The first-conversation chapter](../book/chat.md) — the tools in a
|
|
66
|
+
working chat agent, and each search provider's privacy posture.
|
|
67
|
+
- [The sub-agents chapter](../book/subagents.md) — keeping these
|
|
68
|
+
tools off an agent that reads your files, behind a `RESEARCHER`.
|
|
@@ -0,0 +1,153 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pikuri
|
|
4
|
+
module Web
|
|
5
|
+
# Truncation policy and Tool spec for the +fetch+ tool. HTTP work lives in
|
|
6
|
+
# {Web::Scraper.fetch}; this wrapper accepts only textual content-types,
|
|
7
|
+
# applies a character cap, and exposes the result in OpenAI tool-call shape.
|
|
8
|
+
# Sister of {Web::Scrape} but with no extraction pass — bodies are
|
|
9
|
+
# returned verbatim, for raw textual data (JSON APIs, CSV, +robots.txt+,
|
|
10
|
+
# sitemaps, source files) where a rendering pass would corrupt the payload.
|
|
11
|
+
module Fetch
|
|
12
|
+
# @return [Integer] default character cap on the body from {.fetch}.
|
|
13
|
+
# Smaller than {Web::Scrape::DEFAULT_MAX_CHARS} because fetch's
|
|
14
|
+
# content is bimodal — most JSON/XML/CSV is tiny, and large dumps are
|
|
15
|
+
# better re-requested deliberately than padded into every default.
|
|
16
|
+
DEFAULT_MAX_CHARS = 5_000
|
|
17
|
+
|
|
18
|
+
# @return [Integer] hard ceiling on {.fetch}'s +max_chars+; matches
|
|
19
|
+
# {Web::Scrape::MAX_MAX_CHARS}.
|
|
20
|
+
MAX_MAX_CHARS = 100_000
|
|
21
|
+
|
|
22
|
+
# Application content-types that are textual in practice, so safe to
|
|
23
|
+
# return verbatim despite failing the +text/*+ check. Anything outside
|
|
24
|
+
# +text/*+ and this list is refused.
|
|
25
|
+
# @return [Array<String>]
|
|
26
|
+
TEXTUAL_APPLICATION_TYPES = %w[
|
|
27
|
+
application/json
|
|
28
|
+
application/xml
|
|
29
|
+
application/javascript
|
|
30
|
+
application/xhtml+xml
|
|
31
|
+
application/rss+xml
|
|
32
|
+
application/atom+xml
|
|
33
|
+
].freeze
|
|
34
|
+
|
|
35
|
+
# On-disk cache for {.fetch}, in its own subdir so a +fetch+ and a
|
|
36
|
+
# +web_scrape+ on the same URL don't collide (raw body vs extracted
|
|
37
|
+
# Markdown). A method so specs can swap {UrlCache::NULL}.
|
|
38
|
+
#
|
|
39
|
+
# @return [UrlCache::OnDisk, #fetch]
|
|
40
|
+
CACHE = UrlCache::OnDisk.new(ttl: UrlCache::DEFAULT_TTL, dir: "#{UrlCache::ROOT_DIR}/fetch")
|
|
41
|
+
# Accessor for {CACHE}; specs swap in {UrlCache::NULL} or an isolated cache.
|
|
42
|
+
#
|
|
43
|
+
# @return [UrlCache::OnDisk, #fetch]
|
|
44
|
+
def self.cache
|
|
45
|
+
CACHE
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
# Download +url+ via {Web::Scraper.fetch} and return the body verbatim,
|
|
49
|
+
# if the content-type is textual (any +text/*+ plus
|
|
50
|
+
# {TEXTUAL_APPLICATION_TYPES}); anything else (PDF, image, binary) yields
|
|
51
|
+
# an +"Error: ..."+ string. Cached on disk keyed by URL; +max_chars+ is
|
|
52
|
+
# *not* in the key (values share one entry, truncation runs after the
|
|
53
|
+
# lookup). Cached only on success — {Scraper::FetchError} is caught
|
|
54
|
+
# outside +cache.fetch+, so failures are never persisted; other
|
|
55
|
+
# exceptions bubble up.
|
|
56
|
+
#
|
|
57
|
+
# @param url [String] absolute HTTP(S) URL
|
|
58
|
+
# @param max_chars [Integer] body cap, clamped to +[1, {MAX_MAX_CHARS}]+;
|
|
59
|
+
# default {DEFAULT_MAX_CHARS}. Over the cap, output is cut with a marker.
|
|
60
|
+
# @return [Tool::Result] the body, possibly truncated (+"Fetched 4.2 kB"+),
|
|
61
|
+
# or {Pikuri::Tool::Result.error} on a recoverable failure
|
|
62
|
+
def self.fetch(url, max_chars: DEFAULT_MAX_CHARS)
|
|
63
|
+
max_chars = max_chars.clamp(1, MAX_MAX_CHARS)
|
|
64
|
+
body = cache.fetch(url) { download(url) }
|
|
65
|
+
Tool::Result.new(content: truncate(body, max_chars),
|
|
66
|
+
summary: Scraper.transfer_summary(verb: 'Fetched', full: body, max_chars: max_chars))
|
|
67
|
+
rescue Scraper::FetchError => e
|
|
68
|
+
Tool::Result.error(e.message)
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# GET +url+ and verify the content-type is textual. Always hits the
|
|
72
|
+
# network (caller caches/truncates).
|
|
73
|
+
#
|
|
74
|
+
# @param url [String]
|
|
75
|
+
# @return [String] response body
|
|
76
|
+
# @raise [Scraper::FetchError] on HTTP non-2xx, network failure, redirect
|
|
77
|
+
# exhaustion, missing +Location+, or a non-textual content-type
|
|
78
|
+
def self.download(url)
|
|
79
|
+
fetched = Scraper.fetch(url)
|
|
80
|
+
return fetched.body if textual?(fetched.content_type)
|
|
81
|
+
|
|
82
|
+
raise Scraper::FetchError,
|
|
83
|
+
"refused to fetch #{url}: content-type #{fetched.content_type.inspect} " \
|
|
84
|
+
'is not textual (use web_scrape for rendered pages)'
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
# @param content_type [String] normalized content-type from {Scraper.fetch}
|
|
88
|
+
# @return [Boolean] true when +text/*+ or in {TEXTUAL_APPLICATION_TYPES}
|
|
89
|
+
def self.textual?(content_type)
|
|
90
|
+
content_type.start_with?('text/') ||
|
|
91
|
+
TEXTUAL_APPLICATION_TYPES.include?(content_type)
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
# Cut +body+ to at most +max_chars+, appending a marker with the original
|
|
95
|
+
# length when truncation happens. Same shape as {Web::Scrape.truncate}.
|
|
96
|
+
#
|
|
97
|
+
# @param body [String] full response body
|
|
98
|
+
# @param max_chars [Integer] cap; assumed already clamped
|
|
99
|
+
# @return [String]
|
|
100
|
+
def self.truncate(body, max_chars)
|
|
101
|
+
return body if body.length <= max_chars
|
|
102
|
+
|
|
103
|
+
"#{body[0, max_chars]}\n\n" \
|
|
104
|
+
"... [truncated at #{max_chars} of #{body.length} chars; " \
|
|
105
|
+
'call again with a larger `max_chars` to see more]'
|
|
106
|
+
end
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
# Verbatim URL download tool — a thin wrapper over {Web::Fetch.fetch} in
|
|
110
|
+
# OpenAI tool-call shape. Use for raw textual payloads (JSON, CSV,
|
|
111
|
+
# +robots.txt+, source files); use {Web::SCRAPE} for rendered pages.
|
|
112
|
+
#
|
|
113
|
+
# No confirmer may ever auto-approve a URL that "looks clean": a secret
|
|
114
|
+
# leaves through a bare path (+paste.example/SECRET+), a subdomain (the DNS
|
|
115
|
+
# lookup alone reaches the attacker's nameserver) or a spotless paste-site
|
|
116
|
+
# URL the attacker reads back. Even a confirmer showing the whole URL makes
|
|
117
|
+
# this +human_reviewed+, never safe: the model still picks the host.
|
|
118
|
+
#
|
|
119
|
+
# Sharing: +P_shared_benign+ — no per-agent state, but {Fetch::CACHE} is
|
|
120
|
+
# process-wide (and on disk, so cross-process too): agent B's +fetch+ can
|
|
121
|
+
# be answered from agent A's download within the TTL. See
|
|
122
|
+
# {Pikuri::Web::UrlCache::OnDisk} for the race and why it's harmless.
|
|
123
|
+
#
|
|
124
|
+
# @return [Tool]
|
|
125
|
+
FETCH = Tool.new(
|
|
126
|
+
name: 'fetch',
|
|
127
|
+
description: <<~DESC,
|
|
128
|
+
Downloads the given URL and returns its body verbatim.
|
|
129
|
+
|
|
130
|
+
Usage:
|
|
131
|
+
- Use for raw textual payloads: JSON APIs, CSV files, robots.txt, sitemaps, source files — anywhere a rendering pass would corrupt the data.
|
|
132
|
+
- For rendered HTML pages, use web_scrape — it extracts readable content; fetch returns the raw HTML bytes unchanged.
|
|
133
|
+
- Accepts text/* and common textual application/* types (JSON, XML, JS, XHTML, RSS, Atom). Refuses PDFs, images, and other binaries.
|
|
134
|
+
DESC
|
|
135
|
+
parameters: Tool::Parameters.build { |p|
|
|
136
|
+
p.required_string :url,
|
|
137
|
+
'Absolute URL to download, including the scheme, ' \
|
|
138
|
+
'e.g. "https://example.com/data.json".'
|
|
139
|
+
p.optional_integer :max_chars,
|
|
140
|
+
'Maximum number of characters of the body to ' \
|
|
141
|
+
"return. Defaults to #{Fetch::DEFAULT_MAX_CHARS}; hard-capped at " \
|
|
142
|
+
"#{Fetch::MAX_MAX_CHARS}. When the body is longer than this, " \
|
|
143
|
+
'output is cut and a marker reports the full ' \
|
|
144
|
+
'length.'
|
|
145
|
+
},
|
|
146
|
+
execute: ->(url:, max_chars: Fetch::DEFAULT_MAX_CHARS) {
|
|
147
|
+
Fetch.fetch(url, max_chars: max_chars)
|
|
148
|
+
},
|
|
149
|
+
# As {SCRAPE}: attacker-choosable host, attacker-authored response.
|
|
150
|
+
trifecta_legs: Pikuri::Tool::TrifectaLegs::ASSUMED
|
|
151
|
+
)
|
|
152
|
+
end
|
|
153
|
+
end
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'time'
|
|
4
|
+
|
|
5
|
+
module Pikuri
|
|
6
|
+
module Web
|
|
7
|
+
# Reading policy for the +web_read_cached+ tool: a search result's page
|
|
8
|
+
# text, from the {Search::PageStore} a +page_text+ {Search} filled —
|
|
9
|
+
# never from the page's origin.
|
|
10
|
+
#
|
|
11
|
+
# ReadCached.read('a3f9c21e').content
|
|
12
|
+
# # => "URL: https://docs.ruby-lang.org/en/3.4/Data.html\nTitle: Data\n
|
|
13
|
+
# # Indexed: 2026-11-10T05:22:14Z\n\nclass Data …"
|
|
14
|
+
#
|
|
15
|
+
# Takes a handle, never a URL: a URL would mean a fetch, and the page a
|
|
16
|
+
# model picks is a channel (+D_no_result_scraper+) — let alone a URL it
|
|
17
|
+
# writes. For the same reason links inside a page can't be followed. The
|
|
18
|
+
# text is the engine's snapshot — Brave's in excerpts, possibly older than
|
|
19
|
+
# the live page — so the read says when the engine fetched it, when known.
|
|
20
|
+
#
|
|
21
|
+
# Named +web_+ for the family, +read+ for pikuri's verb for "give me the
|
|
22
|
+
# content", +cached+ because it is no live fetch (+web_cache+ reads as an
|
|
23
|
+
# order to cache something).
|
|
24
|
+
module ReadCached
|
|
25
|
+
# Read the page behind +handle+, cut to +max_chars+ with {Scrape}'s
|
|
26
|
+
# marker.
|
|
27
|
+
#
|
|
28
|
+
# @param handle [String] as printed in the search results, e.g. +"a3f9c21e"+
|
|
29
|
+
# @param max_chars [Integer] text cap, clamped to
|
|
30
|
+
# +[1, {Scrape::MAX_MAX_CHARS}]+; default {Scrape::DEFAULT_MAX_CHARS}
|
|
31
|
+
# @param store [Search::PageStore]
|
|
32
|
+
# @return [Tool::Result] the URL, title, index time and text (+"Read
|
|
33
|
+
# 12.4 kB"+), or {Tool::Result.error} for an unknown or evicted handle
|
|
34
|
+
def self.read(handle, max_chars: Scrape::DEFAULT_MAX_CHARS, store: Search::PageStore.default)
|
|
35
|
+
page = store.read(handle)
|
|
36
|
+
if page.nil?
|
|
37
|
+
return Tool::Result.error(
|
|
38
|
+
"No stored page for handle #{handle.to_s.strip.inspect}. " \
|
|
39
|
+
'Run web_search again and use a handle from its results.'
|
|
40
|
+
)
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
max_chars = max_chars.clamp(1, Scrape::MAX_MAX_CHARS)
|
|
44
|
+
header = "URL: #{page.url}\nTitle: #{page.title}\n"
|
|
45
|
+
header += "Indexed: #{page.indexed_at.utc.iso8601}\n" if page.indexed_at
|
|
46
|
+
Tool::Result.new(content: "#{header}\n#{Scrape.truncate(page.text, max_chars)}",
|
|
47
|
+
summary: Scraper.transfer_summary(verb: 'Read', full: page.text, max_chars: max_chars))
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# A web_search result's stored page text — a thin wrapper over
|
|
52
|
+
# {Web::ReadCached.read} in OpenAI tool-call shape. Pairs with a
|
|
53
|
+
# +page_text+ {Search.build}; wired without one, every read answers
|
|
54
|
+
# "search again".
|
|
55
|
+
#
|
|
56
|
+
# Sends nothing: no network request at all, so the bytes it returns came
|
|
57
|
+
# in with the search, and what left the machine was that search's query —
|
|
58
|
+
# see each engine's +Privacy posture+ ({Search::Brave}, {Search::Exa}).
|
|
59
|
+
#
|
|
60
|
+
# Sharing: +P_shared_locked+ — no state of its own; it reads the
|
|
61
|
+
# process-wide {Search::PageStore}, whose +Mutex+ guards it.
|
|
62
|
+
#
|
|
63
|
+
# @return [Tool]
|
|
64
|
+
READ_CACHED = Tool.new(
|
|
65
|
+
name: 'web_read_cached',
|
|
66
|
+
description: <<~DESC,
|
|
67
|
+
Returns the full text of a web_search result's page.
|
|
68
|
+
|
|
69
|
+
Usage:
|
|
70
|
+
- Takes a handle from web_search's results — the hex code in brackets before a result's title, e.g. [a3f9c21e] — never a URL. A result without a handle can't be read.
|
|
71
|
+
- When a handle is unknown, run web_search again and use a handle from its results.
|
|
72
|
+
- The text may come in excerpts and be older than the live page; the Indexed line, when present, says when it was captured.
|
|
73
|
+
- Links inside the page can't be followed: only pages web_search returned are readable.
|
|
74
|
+
DESC
|
|
75
|
+
parameters: Tool::Parameters.build { |p|
|
|
76
|
+
p.required_string :handle,
|
|
77
|
+
'Handle of a web_search result, as printed in brackets ' \
|
|
78
|
+
'before its title, e.g. "a3f9c21e".'
|
|
79
|
+
p.optional_integer :max_chars,
|
|
80
|
+
'Maximum number of characters of page text to ' \
|
|
81
|
+
"return. Defaults to #{Scrape::DEFAULT_MAX_CHARS}; hard-capped at " \
|
|
82
|
+
"#{Scrape::MAX_MAX_CHARS}. When the page is longer than this, " \
|
|
83
|
+
'output is cut and a marker reports the full ' \
|
|
84
|
+
'length.'
|
|
85
|
+
},
|
|
86
|
+
execute: ->(handle:, max_chars: Scrape::DEFAULT_MAX_CHARS) {
|
|
87
|
+
ReadCached.read(handle, max_chars: max_chars)
|
|
88
|
+
},
|
|
89
|
+
# No request, so no egress leg; the text is whatever the page's author
|
|
90
|
+
# wrote, verbatim into context, hence +:hard+.
|
|
91
|
+
trifecta_legs: Pikuri::Tool::TrifectaLegs.new(
|
|
92
|
+
private: false, untrusted: :hard, egress_payload_review: :no_egress
|
|
93
|
+
)
|
|
94
|
+
)
|
|
95
|
+
end
|
|
96
|
+
end
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pikuri
|
|
4
|
+
module Web
|
|
5
|
+
# Truncation policy and Tool spec for the +web_scrape+ tool. Scraping lives
|
|
6
|
+
# in {Web::Scraper}; this wrapper applies a character cap and exposes the
|
|
7
|
+
# result in OpenAI tool-call shape.
|
|
8
|
+
module Scrape
|
|
9
|
+
# @return [Integer] default character cap on the Markdown from {.visit}.
|
|
10
|
+
# Sized to cover most post-readability article bodies in full on the
|
|
11
|
+
# first call, so the LLM doesn't re-request and duplicate the prefix in
|
|
12
|
+
# its context. Genuinely long pages still get cut, and the marker invites
|
|
13
|
+
# a deliberate larger call.
|
|
14
|
+
DEFAULT_MAX_CHARS = 20_000
|
|
15
|
+
|
|
16
|
+
# @return [Integer] hard ceiling on {.visit}'s +max_chars+; requests above
|
|
17
|
+
# it are clamped silently so the LLM can't dump an arbitrarily large page.
|
|
18
|
+
MAX_MAX_CHARS = 100_000
|
|
19
|
+
|
|
20
|
+
# On-disk cache for {.visit}. A method so specs can swap {UrlCache::NULL}.
|
|
21
|
+
#
|
|
22
|
+
# @return [UrlCache::OnDisk, #fetch]
|
|
23
|
+
CACHE = UrlCache::OnDisk.new(ttl: UrlCache::DEFAULT_TTL, dir: "#{UrlCache::ROOT_DIR}/web_scrape")
|
|
24
|
+
# Accessor for {CACHE}; specs swap in {UrlCache::NULL} or an isolated cache.
|
|
25
|
+
#
|
|
26
|
+
# @return [UrlCache::OnDisk, #fetch]
|
|
27
|
+
def self.cache
|
|
28
|
+
CACHE
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# Fetch +url+ via {Web::Scraper} and truncate the rendered Markdown to
|
|
32
|
+
# +max_chars+. The full Markdown is cached on disk keyed by URL (repeat
|
|
33
|
+
# visits skip the network and extraction); +max_chars+ is *not* in the key
|
|
34
|
+
# (values share one entry, truncation runs after the lookup).
|
|
35
|
+
# {Scraper::FetchError} is caught outside +cache.fetch+ and returned as
|
|
36
|
+
# +"Error: ..."+ (so the LLM can retry a different URL) — never persisted;
|
|
37
|
+
# other exceptions bubble up.
|
|
38
|
+
#
|
|
39
|
+
# @param url [String] absolute HTTP(S) URL
|
|
40
|
+
# @param max_chars [Integer] Markdown cap, clamped to +[1, {MAX_MAX_CHARS}]+;
|
|
41
|
+
# default {DEFAULT_MAX_CHARS}. Over the cap, output is cut with a marker.
|
|
42
|
+
# @return [Tool::Result] the Markdown, possibly truncated (+"Scraped 12.4
|
|
43
|
+
# kB"+), or {Pikuri::Tool::Result.error} on a recoverable fetch failure
|
|
44
|
+
def self.visit(url, max_chars: DEFAULT_MAX_CHARS)
|
|
45
|
+
max_chars = max_chars.clamp(1, MAX_MAX_CHARS)
|
|
46
|
+
markdown = cache.fetch(url) { Scraper.visit(url) }
|
|
47
|
+
Tool::Result.new(content: truncate(markdown, max_chars),
|
|
48
|
+
summary: Scraper.transfer_summary(verb: 'Scraped', full: markdown,
|
|
49
|
+
max_chars: max_chars))
|
|
50
|
+
rescue Scraper::FetchError => e
|
|
51
|
+
Tool::Result.error(e.message)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# Cut +markdown+ to at most +max_chars+, appending a marker with the
|
|
55
|
+
# original length when truncation happens.
|
|
56
|
+
#
|
|
57
|
+
# @param markdown [String] full Markdown text
|
|
58
|
+
# @param max_chars [Integer] cap; assumed already clamped
|
|
59
|
+
# @return [String]
|
|
60
|
+
def self.truncate(markdown, max_chars)
|
|
61
|
+
return markdown if markdown.length <= max_chars
|
|
62
|
+
|
|
63
|
+
"#{markdown[0, max_chars]}\n\n" \
|
|
64
|
+
"... [truncated at #{max_chars} of #{markdown.length} chars; " \
|
|
65
|
+
'call again with a larger `max_chars` to see more]'
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# Webpage download + Markdown conversion tool — a thin wrapper over
|
|
70
|
+
# {Web::Scrape.visit} in OpenAI tool-call shape.
|
|
71
|
+
#
|
|
72
|
+
# Sharing: +P_shared_benign+ — as {Web::FETCH}: no per-agent state, but
|
|
73
|
+
# {Scrape::CACHE} is process-wide and on disk, so one agent's scrape
|
|
74
|
+
# answers another's within the TTL. See {Pikuri::Web::UrlCache::OnDisk}.
|
|
75
|
+
#
|
|
76
|
+
# @return [Tool]
|
|
77
|
+
SCRAPE = Tool.new(
|
|
78
|
+
name: 'web_scrape',
|
|
79
|
+
description: <<~DESC,
|
|
80
|
+
Scrapes the rendered webpage or text file at the given URL and returns its main content as Markdown.
|
|
81
|
+
|
|
82
|
+
Usage:
|
|
83
|
+
- Use for HTML pages where you want readable content — readability extraction strips nav, sidebars, and boilerplate.
|
|
84
|
+
- For raw textual payloads (JSON, CSV, robots.txt, source files), use fetch instead — it returns bytes verbatim, while web_scrape would corrupt them with a Markdown pass.
|
|
85
|
+
- A Single Page App may return very little or no content. Do NOT retry with a larger max_chars; try a different URL instead.
|
|
86
|
+
DESC
|
|
87
|
+
parameters: Tool::Parameters.build { |p|
|
|
88
|
+
p.required_string :url,
|
|
89
|
+
'Absolute URL of the webpage to scrape, including ' \
|
|
90
|
+
'the scheme, e.g. "https://example.com/article".'
|
|
91
|
+
p.optional_integer :max_chars,
|
|
92
|
+
'Maximum number of characters of Markdown to ' \
|
|
93
|
+
"return. Defaults to #{Scrape::DEFAULT_MAX_CHARS}; hard-capped at " \
|
|
94
|
+
"#{Scrape::MAX_MAX_CHARS}. When the page is longer than this, " \
|
|
95
|
+
'output is cut and a marker reports the full ' \
|
|
96
|
+
'length.'
|
|
97
|
+
},
|
|
98
|
+
execute: ->(url:, max_chars: Scrape::DEFAULT_MAX_CHARS) {
|
|
99
|
+
Scrape.visit(url, max_chars: max_chars)
|
|
100
|
+
},
|
|
101
|
+
# Exactly {TrifectaLegs::ASSUMED}, and the archetype it was named for: the
|
|
102
|
+
# page is authored by whoever runs the site, and the model picks the URL,
|
|
103
|
+
# so the request is itself an outbound channel.
|
|
104
|
+
trifecta_legs: Pikuri::Tool::TrifectaLegs::ASSUMED
|
|
105
|
+
)
|
|
106
|
+
end
|
|
107
|
+
end
|
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'faraday'
|
|
4
|
+
require 'stringio'
|
|
5
|
+
require 'uri'
|
|
6
|
+
|
|
7
|
+
module Pikuri
|
|
8
|
+
module Web
|
|
9
|
+
# HTTP side of the web tools ({Web::SCRAPE} and {Web::FETCH}): GET the
|
|
10
|
+
# URL with a real-browser User-Agent, follow redirects, and hand the body to
|
|
11
|
+
# {Pikuri::Extractor.extract} with the response +Content-Type+ as the hint.
|
|
12
|
+
# HTML/XHTML renders via {Pikuri::Extractors::HTML}, other +text/*+ passes through
|
|
13
|
+
# verbatim, plug-in extractors extend the set (with pikuri-pdf,
|
|
14
|
+
# +application/pdf+ extracts — by header or +%PDF-+ magic, so a lying header
|
|
15
|
+
# still works); remaining types raise {FetchError} so the LLM observes the
|
|
16
|
+
# failure rather than an empty rendering.
|
|
17
|
+
#
|
|
18
|
+
# Split into a thin HTTP fetch ({.fetch}) and the extraction wrapper
|
|
19
|
+
# ({.visit}) so tests drive each in isolation and {Web::Fetch} can reuse the
|
|
20
|
+
# HTTP half. Nothing here knows about the LLM — the wrapping tools own
|
|
21
|
+
# caching, truncation, and turning the result into an observation.
|
|
22
|
+
module Scraper
|
|
23
|
+
# Raised when a URL can't be rendered to Markdown — HTTP non-2xx, network
|
|
24
|
+
# failure, redirect loop, missing +Location+, unsupported content-type, or
|
|
25
|
+
# a parse failure. The web tools rescue this into an +"Error: ..."+
|
|
26
|
+
# observation; anything else bubbles up so genuine bugs stay visible.
|
|
27
|
+
class FetchError < StandardError; end
|
|
28
|
+
|
|
29
|
+
# @return [String] User-Agent sent with each request: Firefox ESR on
|
|
30
|
+
# Linux. Sites reject a bot-looking UA and some a stale one (technical.city
|
|
31
|
+
# drops +Chrome/120+), so bump it to each new ESR, roughly yearly. ESR,
|
|
32
|
+
# not rapid release: a rapid-release version goes stale in weeks, on a
|
|
33
|
+
# cadence Mozilla keeps changing.
|
|
34
|
+
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0'
|
|
35
|
+
# @return [String] +Accept+ header, so content-negotiating servers hand
|
|
36
|
+
# back something usable: HTML first, then +application/pdf+, then +text/*+.
|
|
37
|
+
ACCEPT = 'text/html,application/xhtml+xml,application/pdf,text/*;q=0.8'
|
|
38
|
+
# @return [Integer] maximum HTTP redirects to follow before giving up.
|
|
39
|
+
MAX_REDIRECTS = 5
|
|
40
|
+
# @return [Integer] connect timeout (s) for the Faraday request.
|
|
41
|
+
OPEN_TIMEOUT = 10
|
|
42
|
+
# @return [Integer] read timeout (s) for the Faraday request.
|
|
43
|
+
READ_TIMEOUT = 20
|
|
44
|
+
|
|
45
|
+
# @return [Integer] max chars of an error response body in a {FetchError}
|
|
46
|
+
# message — enough to show what came back (a Cloudflare/WAF page) without
|
|
47
|
+
# flooding the observation.
|
|
48
|
+
ERROR_BODY_EXCERPT = 200
|
|
49
|
+
|
|
50
|
+
# Result of a successful {Scraper.fetch}: the response body, the
|
|
51
|
+
# normalized content-type (lower-cased, with any +; charset=...+
|
|
52
|
+
# parameters stripped), and the final URL after redirects.
|
|
53
|
+
Fetched = Data.define(:body, :content_type, :url)
|
|
54
|
+
|
|
55
|
+
# Fetch +url+ and render its main content as Markdown. No caching — every
|
|
56
|
+
# call hits the network (wrap it to memoize; {Web::Scrape.visit}
|
|
57
|
+
# does). The output is +strip+'d (leading/trailing blank lines are common
|
|
58
|
+
# from PDF page-feeds and trailing newlines) but interior whitespace is
|
|
59
|
+
# preserved (paragraph breaks and code indentation are load-bearing).
|
|
60
|
+
#
|
|
61
|
+
# @param url [String] absolute HTTP(S) URL
|
|
62
|
+
# @return [String] Markdown, whitespace-trimmed, otherwise uncapped —
|
|
63
|
+
# caller size-limits before feeding the LLM
|
|
64
|
+
# @raise [FetchError] on HTTP non-2xx, network failure, redirect loop, a
|
|
65
|
+
# 3xx without +Location+, an unrecognized response, or extraction failure
|
|
66
|
+
def self.visit(url)
|
|
67
|
+
extract(fetch(url)).strip
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
# Render a {Fetched} response as Markdown via {Pikuri::Extractor.extract},
|
|
71
|
+
# re-raising both extraction failure modes as {FetchError}. The
|
|
72
|
+
# content-type is passed verbatim (including +""+ for a missing header,
|
|
73
|
+
# which matches no text arm — a body without transport metadata is
|
|
74
|
+
# refused, not sniffed; only a strong magic sniff like +%PDF-+ overrides).
|
|
75
|
+
#
|
|
76
|
+
# @param fetched [Fetched]
|
|
77
|
+
# @return [String] Markdown from the matched extractor
|
|
78
|
+
# @raise [FetchError] when no extractor matches, or extraction fails
|
|
79
|
+
def self.extract(fetched)
|
|
80
|
+
Pikuri::Extractor.extract(StringIO.new(fetched.body), content_type: fetched.content_type)
|
|
81
|
+
rescue Pikuri::Extractor::Unsupported
|
|
82
|
+
raise FetchError, "unsupported content-type #{fetched.content_type.inspect} for #{fetched.url}"
|
|
83
|
+
rescue Pikuri::Extractor::Error => e
|
|
84
|
+
raise FetchError, e.message
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
# Download +url+, manually following up to {MAX_REDIRECTS} redirects. All
|
|
88
|
+
# recoverable failures (HTTP 4xx/5xx, +Faraday::Error+, exhausted redirect
|
|
89
|
+
# budget, 3xx without +Location+) surface as {FetchError}. Error bodies
|
|
90
|
+
# are trimmed to {ERROR_BODY_EXCERPT} so a challenge page doesn't dump
|
|
91
|
+
# kilobytes into the observation.
|
|
92
|
+
#
|
|
93
|
+
# @param url [String] absolute HTTP(S) URL
|
|
94
|
+
# @param limit [Integer] redirects remaining; recurses with +limit - 1+ on 3xx
|
|
95
|
+
# @return [Fetched] body, normalized content-type, final URL
|
|
96
|
+
# @raise [FetchError] on non-2xx/3xx, network error, redirect-loop
|
|
97
|
+
# exhaustion, or 3xx without +Location+
|
|
98
|
+
def self.fetch(url, limit: MAX_REDIRECTS)
|
|
99
|
+
raise FetchError, "too many redirects fetching #{url}" if limit.zero?
|
|
100
|
+
|
|
101
|
+
response = begin
|
|
102
|
+
browser_connection(url).get(url) { |req| req.headers['Accept'] = ACCEPT }
|
|
103
|
+
rescue Faraday::Error => e
|
|
104
|
+
raise FetchError, "#{e.class.name.split('::').last} fetching #{url}: #{e.message}"
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
case response.status
|
|
108
|
+
when 200..299
|
|
109
|
+
Fetched.new(body: response.body, content_type: normalize_content_type(response.headers['content-type']), url: url)
|
|
110
|
+
when 300..399
|
|
111
|
+
location = response.headers['location']
|
|
112
|
+
raise FetchError, "HTTP #{response.status} from #{url} with no Location header" if location.nil? || location.empty?
|
|
113
|
+
|
|
114
|
+
fetch(URI.join(url, location).to_s, limit: limit - 1)
|
|
115
|
+
else
|
|
116
|
+
raise FetchError, "HTTP #{response.status} fetching #{url}: #{excerpt(response.body)}"
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# A Faraday connection that imitates a browser — Firefox — to +url+'s host:
|
|
121
|
+
# +Host+ first, then {USER_AGENT}, with the {OPEN_TIMEOUT} / {READ_TIMEOUT}
|
|
122
|
+
# budget. For a page meant for people in a browser, which a site defends
|
|
123
|
+
# against bots (web pages, DuckDuckGo's HTML results):
|
|
124
|
+
#
|
|
125
|
+
# Scraper.browser_connection(ENDPOINT).get(ENDPOINT, { q: query })
|
|
126
|
+
#
|
|
127
|
+
# Not for an API meant for programs (Brave's, Exa's): call plain Faraday
|
|
128
|
+
# there, as the client pikuri is — imitating a browser buys nothing.
|
|
129
|
+
# Headers set on a request land after these two, keeping the browser order.
|
|
130
|
+
#
|
|
131
|
+
# @param url [String] absolute HTTP(S) URL; its host and port become +Host+
|
|
132
|
+
# @return [Faraday::Connection] good for that one host only
|
|
133
|
+
def self.browser_connection(url)
|
|
134
|
+
uri = URI(url)
|
|
135
|
+
# Host first, as browsers send it: Net::HTTP would append it last, and a
|
|
136
|
+
# browser UA with a trailing Host draws Cloudflare's 403 challenge.
|
|
137
|
+
headers = {
|
|
138
|
+
'Host' => uri.port == uri.default_port ? uri.host : "#{uri.host}:#{uri.port}",
|
|
139
|
+
'User-Agent' => USER_AGENT
|
|
140
|
+
}
|
|
141
|
+
Faraday.new(headers: headers, request: { open_timeout: OPEN_TIMEOUT, timeout: READ_TIMEOUT })
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
# Display line for a downloaded body — what the model got, plus the full
|
|
145
|
+
# size when the cap cut it:
|
|
146
|
+
#
|
|
147
|
+
# Scraper.transfer_summary(verb: 'Fetched', full: body, max_chars: 5_000)
|
|
148
|
+
# # => "Fetched 5.0 kB of 41.2 kB"
|
|
149
|
+
#
|
|
150
|
+
# No content-type in it: both wrappers cache the *body*, so a cache hit
|
|
151
|
+
# has no header to report, and a line that came and went with the cache
|
|
152
|
+
# would be worse than one that never names the type.
|
|
153
|
+
#
|
|
154
|
+
# @param verb [String] +"Fetched"+ or +"Scraped"+
|
|
155
|
+
# @param full [String] the whole body, before truncation
|
|
156
|
+
# @param max_chars [Integer] the cap applied to it
|
|
157
|
+
# @return [String]
|
|
158
|
+
def self.transfer_summary(verb:, full:, max_chars:)
|
|
159
|
+
return "#{verb} #{Pikuri::Tool.format_bytes(full.bytesize)}" if full.length <= max_chars
|
|
160
|
+
|
|
161
|
+
"#{verb} #{Pikuri::Tool.format_bytes(full[0, max_chars].bytesize)} " \
|
|
162
|
+
"of #{Pikuri::Tool.format_bytes(full.bytesize)}"
|
|
163
|
+
end
|
|
164
|
+
|
|
165
|
+
# Lower-case +raw+ and strip any +; charset=...+ parameters so the
|
|
166
|
+
# extractors can match on a canonical token.
|
|
167
|
+
#
|
|
168
|
+
# @param raw [String, nil] raw +Content-Type+ header value
|
|
169
|
+
# @return [String] normalized content-type, or +""+ when the
|
|
170
|
+
# header was missing
|
|
171
|
+
def self.normalize_content_type(raw)
|
|
172
|
+
raw.to_s.split(';').first.to_s.strip.downcase
|
|
173
|
+
end
|
|
174
|
+
private_class_method :normalize_content_type
|
|
175
|
+
|
|
176
|
+
# Whitespace-collapse +body+ and clip to {ERROR_BODY_EXCERPT}, keeping the
|
|
177
|
+
# {FetchError} message a single readable line.
|
|
178
|
+
#
|
|
179
|
+
# @param body [String, nil]
|
|
180
|
+
# @return [String]
|
|
181
|
+
def self.excerpt(body)
|
|
182
|
+
text = body.to_s.gsub(/\s+/, ' ').strip
|
|
183
|
+
text.length > ERROR_BODY_EXCERPT ? "#{text[0, ERROR_BODY_EXCERPT]}..." : text
|
|
184
|
+
end
|
|
185
|
+
private_class_method :excerpt
|
|
186
|
+
end
|
|
187
|
+
end
|
|
188
|
+
end
|