ask-web-fetch 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +45 -14
- data/lib/ask/web_fetch/backend.rb +66 -0
- data/lib/ask/web_fetch/backends/jina.rb +53 -0
- data/lib/ask/web_fetch/backends/local.rb +101 -0
- data/lib/ask/web_fetch/tool.rb +30 -74
- data/lib/ask/web_fetch/version.rb +1 -1
- data/lib/ask/web_fetch.rb +3 -0
- metadata +8 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 33eb9a14775223cfaf64bab5b01544c8eb79bb9f4b28c2c8a414bd579116e1e7
|
|
4
|
+
data.tar.gz: b75238a2a93d2f04c8d7327a993c2a097ab09df3169ce005e67c32b4d25ba00f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 380b32b742ef62153c8def11dc619e133d1230d7ec39c73b7787448550d5d3f8b37ee9fbe7b5e09a886ba29e7cbfefbdbe0f73d4999255bd4cd19202c03f12a8
|
|
7
|
+
data.tar.gz: a0f520160d28b978adae2a52414b2625cb88f57bec7a1cab5139277db7f5d639410889915e7a2fbfebf803082cdb5470a48b145ace49776237a91ab8be302e4d
|
data/README.md
CHANGED
|
@@ -4,19 +4,47 @@
|
|
|
4
4
|
|
|
5
5
|
A web fetch tool for the ask-rb ecosystem. It provides
|
|
6
6
|
`Ask::Tools::WebFetch`, which fetches a URL and returns its content as clean
|
|
7
|
-
markdown for LLM consumption. It has no Rails dependencies
|
|
8
|
-
|
|
9
|
-
reverse_markdown).
|
|
7
|
+
markdown for LLM consumption. It has no Rails dependencies and no API key
|
|
8
|
+
required.
|
|
10
9
|
|
|
11
10
|
## How it works
|
|
12
11
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
12
|
+
`Ask::Tools::WebFetch` runs a chain of pluggable backends and returns the
|
|
13
|
+
first success:
|
|
14
|
+
|
|
15
|
+
1. **Local** (default) — pure Ruby `Net::HTTP` + Nokogiri + reverse_markdown:
|
|
16
|
+
browser-like User-Agent, redirects followed, main content extracted
|
|
17
|
+
(`<article>` → `<main>` → `<body>`), navigation/scripts stripped, tables
|
|
18
|
+
become markdown tables, links become `[text](url)`.
|
|
19
|
+
2. **Jina** — Jina Reader free tier (`https://r.jina.ai/<url>`). It runs
|
|
20
|
+
headless Chromium, so it renders JS pages the Local backend can't. Free
|
|
21
|
+
without a key (~20 req/min per IP); set `JINA_API_KEY` for higher limits.
|
|
22
|
+
|
|
23
|
+
The tool falls back automatically: if Local fails (blocked, timeout,
|
|
24
|
+
non-HTML, anti-bot challenge, or a JS page with no server-side content), it
|
|
25
|
+
tries Jina. If Jina fails too (rate limit, access error, challenge page),
|
|
26
|
+
the call returns a failure result listing each backend's error.
|
|
27
|
+
|
|
28
|
+
### Adding a backend
|
|
29
|
+
|
|
30
|
+
Backends subclass `Ask::WebFetch::Backend` and implement one method:
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
class Crawl4ai < Ask::WebFetch::Backend
|
|
34
|
+
def fetch(url)
|
|
35
|
+
# return { title: "Page Title", content: "markdown..." }
|
|
36
|
+
# or raise Ask::WebFetch::FetchError / EmptyContentError
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
Ask::Tools::WebFetch.backends = [Crawl4ai, Ask::WebFetch::Backends::Local]
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
`#fetch` must return `{ title: String|nil, content: String }` and raise
|
|
44
|
+
`Ask::WebFetch::FetchError` (hard failure) or `EmptyContentError` (page
|
|
45
|
+
fetched but nothing usable). The chain then handles ordering and fallback
|
|
46
|
+
for you. For tests, `Ask::Tools::WebFetch.backends = [...]` can be swapped
|
|
47
|
+
and restored.
|
|
20
48
|
|
|
21
49
|
## Installation
|
|
22
50
|
|
|
@@ -51,13 +79,16 @@ You can cap the output size:
|
|
|
51
79
|
result = tool.execute(url: "https://example.com", max_chars: 5000)
|
|
52
80
|
```
|
|
53
81
|
|
|
54
|
-
If the page
|
|
55
|
-
|
|
82
|
+
If every backend fails (e.g. the page is behind an anti-bot challenge and
|
|
83
|
+
Jina is also rate-limited), the call returns a failure result with each
|
|
84
|
+
backend's error.
|
|
56
85
|
|
|
57
86
|
## Configuration
|
|
58
87
|
|
|
59
|
-
No configuration required
|
|
60
|
-
|
|
88
|
+
No configuration required for the default chain. Optional knobs:
|
|
89
|
+
|
|
90
|
+
- `JINA_API_KEY` — enables the Jina fallback with higher rate limits
|
|
91
|
+
- `max_chars` parameter — caps output length (default 20000)
|
|
61
92
|
|
|
62
93
|
## Known limitations
|
|
63
94
|
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'version'
|
|
4
|
+
|
|
5
|
+
module Ask
|
|
6
|
+
module WebFetch
|
|
7
|
+
# Raised by backends on any failure; the tool catches it and tries the
|
|
8
|
+
# next backend in the chain.
|
|
9
|
+
class Error < StandardError; end
|
|
10
|
+
|
|
11
|
+
# A backend that failed to fetch (network error, non-2xx, challenge
|
|
12
|
+
# page, non-HTML response).
|
|
13
|
+
class FetchError < Error; end
|
|
14
|
+
|
|
15
|
+
# A backend that fetched the page but found nothing usable in it.
|
|
16
|
+
class EmptyContentError < Error; end
|
|
17
|
+
|
|
18
|
+
# Base class for fetch backends, plus the errors they raise.
|
|
19
|
+
#
|
|
20
|
+
# A backend turns a URL into LLM-ready markdown. To add a new backend:
|
|
21
|
+
#
|
|
22
|
+
# 1. subclass Backend and implement #fetch(url)
|
|
23
|
+
# 2. #fetch must return { title: String|nil, content: String }
|
|
24
|
+
# 3. #fetch must raise FetchError (hard failure) or
|
|
25
|
+
# EmptyContentError (page fetched but nothing usable) on failure
|
|
26
|
+
# 4. register the class in Ask::Tools::WebFetch.backends
|
|
27
|
+
#
|
|
28
|
+
# The tool tries each backend in order and returns the first success.
|
|
29
|
+
class Backend
|
|
30
|
+
# Identity sent on every request, browser-like plus a gem tag.
|
|
31
|
+
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
|
|
32
|
+
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 ' \
|
|
33
|
+
"ask-web-fetch/#{Ask::WebFetch::VERSION}".freeze
|
|
34
|
+
|
|
35
|
+
# Content shorter than this is treated as a page with no usable
|
|
36
|
+
# content (e.g. a JS-rendered shell with nothing server-side).
|
|
37
|
+
MIN_CONTENT_LENGTH = 100
|
|
38
|
+
|
|
39
|
+
# Cloudflare-style anti-bot signatures. Deliberately narrow:
|
|
40
|
+
# challenge/interstitial pages carry these markers, while legitimate
|
|
41
|
+
# pages can contain the word "captcha" in unrelated config/JS (e.g.
|
|
42
|
+
# Wikipedia embeds an hcaptcha edit-config flag on every page).
|
|
43
|
+
CHALLENGE_RE = /just a moment|checking your browser|cf-chl/i
|
|
44
|
+
|
|
45
|
+
def self.backend_name
|
|
46
|
+
name.split('::').last
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
# Fetches +url+ and returns { title: String|nil, content: String }.
|
|
50
|
+
# Raises FetchError or EmptyContentError on failure.
|
|
51
|
+
def fetch(url)
|
|
52
|
+
raise NotImplementedError, "#{self.class} must implement #fetch(url)"
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
private
|
|
56
|
+
|
|
57
|
+
def challenge_page?(body)
|
|
58
|
+
body.to_s.match?(CHALLENGE_RE)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def usable_content?(content)
|
|
62
|
+
content.to_s.strip.length >= MIN_CONTENT_LENGTH
|
|
63
|
+
end
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
end
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'net/http'
|
|
4
|
+
require 'uri'
|
|
5
|
+
require_relative '../backend'
|
|
6
|
+
|
|
7
|
+
module Ask
|
|
8
|
+
module WebFetch
|
|
9
|
+
module Backends
|
|
10
|
+
# Jina Reader free tier: GET https://r.jina.ai/<url>.
|
|
11
|
+
#
|
|
12
|
+
# Free without a key (~20 req/min per IP). Set JINA_API_KEY for higher
|
|
13
|
+
# rate limits. The endpoint runs headless Chromium, so it renders JS
|
|
14
|
+
# pages that the Local backend cannot.
|
|
15
|
+
class Jina < Backend
|
|
16
|
+
BASE_URL = 'https://r.jina.ai'
|
|
17
|
+
OPEN_TIMEOUT = 5
|
|
18
|
+
READ_TIMEOUT = 30
|
|
19
|
+
|
|
20
|
+
def fetch(url)
|
|
21
|
+
uri = URI("#{BASE_URL}/#{url}")
|
|
22
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
23
|
+
http.use_ssl = true
|
|
24
|
+
http.open_timeout = OPEN_TIMEOUT
|
|
25
|
+
http.read_timeout = READ_TIMEOUT
|
|
26
|
+
req = Net::HTTP::Get.new(uri)
|
|
27
|
+
req['User-Agent'] = USER_AGENT
|
|
28
|
+
req['Accept'] = 'text/markdown, text/plain, text/html'
|
|
29
|
+
req['Authorization'] = "Bearer #{ENV['JINA_API_KEY']}" if ENV['JINA_API_KEY']
|
|
30
|
+
|
|
31
|
+
res = http.request(req)
|
|
32
|
+
case res.code
|
|
33
|
+
when '200'
|
|
34
|
+
body = res.body.to_s
|
|
35
|
+
raise FetchError, 'challenge page from Jina' if challenge_page?(body)
|
|
36
|
+
raise EmptyContentError, 'empty response from Jina' unless usable_content?(body)
|
|
37
|
+
|
|
38
|
+
{ title: nil, content: body.strip }
|
|
39
|
+
when '429'
|
|
40
|
+
raise FetchError, 'rate limited by Jina (429)'
|
|
41
|
+
when '401', '403'
|
|
42
|
+
raise FetchError, "Jina access error (#{res.code})"
|
|
43
|
+
else
|
|
44
|
+
raise FetchError, "Jina returned #{res.code}"
|
|
45
|
+
end
|
|
46
|
+
rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
|
|
47
|
+
Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
|
|
48
|
+
raise FetchError, "Jina #{e.class}: #{e.message}"
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'net/http'
|
|
4
|
+
require 'uri'
|
|
5
|
+
require 'nokogiri'
|
|
6
|
+
require 'reverse_markdown'
|
|
7
|
+
require_relative '../backend'
|
|
8
|
+
|
|
9
|
+
module Ask
|
|
10
|
+
module WebFetch
|
|
11
|
+
module Backends
|
|
12
|
+
# Default backend: pure Ruby Net::HTTP + Nokogiri + reverse_markdown.
|
|
13
|
+
# No external service or API key; mirrors ask-web-search's self-hosted
|
|
14
|
+
# SearXNG approach.
|
|
15
|
+
class Local < Backend
|
|
16
|
+
MAX_REDIRECTS = 5
|
|
17
|
+
OPEN_TIMEOUT = 5
|
|
18
|
+
READ_TIMEOUT = 15
|
|
19
|
+
|
|
20
|
+
# Class/id fragments that mark navigation chrome worth dropping, e.g.
|
|
21
|
+
# "vector-page-toolbar", "sidebar", "toc".
|
|
22
|
+
NAV_CHROME_RE = /
|
|
23
|
+
(^|[\s_-])(nav|menu|toolbar|breadcrumb|sidebar|toc|footer|header|
|
|
24
|
+
banner|pagination|search|cookie|modal|popup)([\s_-]|$)
|
|
25
|
+
/ix
|
|
26
|
+
|
|
27
|
+
def fetch(url)
|
|
28
|
+
body, content_type = fetch_html(url)
|
|
29
|
+
raise FetchError, "expected HTML from #{url}, got #{content_type}" unless content_type.include?('html')
|
|
30
|
+
raise FetchError, "challenge page at #{url}" if challenge_page?(body)
|
|
31
|
+
|
|
32
|
+
page = to_markdown(body, url)
|
|
33
|
+
raise EmptyContentError, "no readable content at #{url}" unless usable_content?(page[:content])
|
|
34
|
+
|
|
35
|
+
page
|
|
36
|
+
rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
|
|
37
|
+
Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
|
|
38
|
+
raise FetchError, "#{e.class}: #{e.message}"
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# Parses +html+ and returns { title:, content: } where content is
|
|
42
|
+
# clean markdown.
|
|
43
|
+
def to_markdown(html, _url)
|
|
44
|
+
doc = Nokogiri::HTML(html)
|
|
45
|
+
candidate = extract_main(doc)
|
|
46
|
+
scrub(candidate)
|
|
47
|
+
markdown = ReverseMarkdown.convert(candidate.to_html, unknown_tags: :bypass, github_flavored: true)
|
|
48
|
+
markdown = clean(markdown)
|
|
49
|
+
title = doc.at('title')&.text&.strip
|
|
50
|
+
{ title: title, content: markdown }
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
private
|
|
54
|
+
|
|
55
|
+
# GET with redirect following (max MAX_REDIRECTS hops).
|
|
56
|
+
def fetch_html(url)
|
|
57
|
+
uri = URI(url)
|
|
58
|
+
hops = 0
|
|
59
|
+
loop do
|
|
60
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
61
|
+
http.open_timeout = OPEN_TIMEOUT
|
|
62
|
+
http.read_timeout = READ_TIMEOUT
|
|
63
|
+
http.use_ssl = uri.scheme == 'https'
|
|
64
|
+
req = Net::HTTP::Get.new(uri)
|
|
65
|
+
req['User-Agent'] = USER_AGENT
|
|
66
|
+
req['Accept'] = 'text/html,application/xhtml+xml'
|
|
67
|
+
res = http.request(req)
|
|
68
|
+
return [res.body, res['content-type'].to_s] if res.code.start_with?('2')
|
|
69
|
+
|
|
70
|
+
raise FetchError, "got #{res.code} from #{url}" unless res.code.start_with?('3') && res['location']
|
|
71
|
+
raise FetchError, "hit a redirect loop at #{url}" if (hops += 1) > MAX_REDIRECTS
|
|
72
|
+
|
|
73
|
+
uri = URI.join(uri, res['location'])
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def extract_main(doc)
|
|
78
|
+
doc.at('article') || doc.at('main') || doc.at('[role="main"]') || doc.at('body') || doc
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
# Remove chrome INSIDE the candidate only, never ancestors.
|
|
82
|
+
def scrub(candidate)
|
|
83
|
+
candidate.css('script, style, noscript, nav, footer, header, iframe, form, svg, aside').each(&:remove)
|
|
84
|
+
candidate.css('*[id], *[class]').each do |el|
|
|
85
|
+
next if el.equal?(candidate)
|
|
86
|
+
|
|
87
|
+
id_cls = [el['id'], el['class']].compact.join(' ')
|
|
88
|
+
el.remove if id_cls.match?(NAV_CHROME_RE)
|
|
89
|
+
end
|
|
90
|
+
candidate
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def clean(markdown)
|
|
94
|
+
markdown.gsub(/[ \t]+\n/, "\n")
|
|
95
|
+
.gsub(/\n{3,}/, "\n\n")
|
|
96
|
+
.strip
|
|
97
|
+
end
|
|
98
|
+
end
|
|
99
|
+
end
|
|
100
|
+
end
|
|
101
|
+
end
|
data/lib/ask/web_fetch/tool.rb
CHANGED
|
@@ -1,31 +1,29 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'ask-tools'
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
require 'reverse_markdown'
|
|
4
|
+
require_relative '../web_fetch/backend'
|
|
5
|
+
require_relative '../web_fetch/backends/local'
|
|
6
|
+
require_relative '../web_fetch/backends/jina'
|
|
8
7
|
|
|
9
8
|
module Ask
|
|
10
9
|
module Tools
|
|
11
10
|
# Fetches a URL and returns its content as clean markdown for LLM
|
|
12
|
-
# consumption.
|
|
13
|
-
#
|
|
11
|
+
# consumption. Tries each configured backend in order and returns the
|
|
12
|
+
# first success, so a blocked or JS-rendered page falls through from the
|
|
13
|
+
# local fetcher to Jina Reader.
|
|
14
14
|
class WebFetch < Ask::Tool
|
|
15
|
-
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
|
|
16
|
-
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 ' \
|
|
17
|
-
"ask-web-fetch/#{Ask::WebFetch::VERSION}".freeze
|
|
18
15
|
DEFAULT_MAX_CHARS = 20_000
|
|
19
|
-
MAX_REDIRECTS = 5
|
|
20
|
-
OPEN_TIMEOUT = 5
|
|
21
|
-
READ_TIMEOUT = 15
|
|
22
16
|
|
|
23
|
-
#
|
|
24
|
-
#
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
17
|
+
# Backend chain, tried in order. Swap or extend for future backends
|
|
18
|
+
# (e.g. a self-hosted crawler); each must subclass
|
|
19
|
+
# Ask::WebFetch::Backend and implement #fetch(url).
|
|
20
|
+
def self.backends
|
|
21
|
+
@backends ||= [Ask::WebFetch::Backends::Local, Ask::WebFetch::Backends::Jina]
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
class << self
|
|
25
|
+
attr_writer :backends
|
|
26
|
+
end
|
|
29
27
|
|
|
30
28
|
description 'Fetch a URL and return its content as clean markdown for LLM consumption. ' \
|
|
31
29
|
'Use this to read web pages, articles, and documentation.'
|
|
@@ -40,74 +38,32 @@ module Ask
|
|
|
40
38
|
)
|
|
41
39
|
|
|
42
40
|
def execute(url:, max_chars: DEFAULT_MAX_CHARS)
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
markdown = to_markdown(body, url)
|
|
41
|
+
page = fetch_page(url)
|
|
42
|
+
markdown = format(page, url)
|
|
47
43
|
markdown = truncate(markdown, max_chars) if max_chars&.positive?
|
|
48
44
|
Ask::Result.ok(data: markdown)
|
|
49
45
|
end
|
|
50
46
|
|
|
51
47
|
private
|
|
52
48
|
|
|
53
|
-
#
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
http.open_timeout = OPEN_TIMEOUT
|
|
61
|
-
http.read_timeout = READ_TIMEOUT
|
|
62
|
-
http.use_ssl = uri.scheme == 'https'
|
|
63
|
-
req = Net::HTTP::Get.new(uri)
|
|
64
|
-
req['User-Agent'] = USER_AGENT
|
|
65
|
-
req['Accept'] = 'text/html,application/xhtml+xml'
|
|
66
|
-
res = http.request(req)
|
|
67
|
-
return [res.body, res['content-type'].to_s] if res.code.start_with?('2')
|
|
68
|
-
|
|
69
|
-
raise "WebFetch got #{res.code} from #{url}" unless res.code.start_with?('3') && res['location']
|
|
70
|
-
raise "WebFetch hit a redirect loop at #{url}" if (hops += 1) > MAX_REDIRECTS
|
|
71
|
-
|
|
72
|
-
uri = URI.join(uri, res['location'])
|
|
49
|
+
# Try each configured backend in order; return the first success.
|
|
50
|
+
def fetch_page(url)
|
|
51
|
+
failures = []
|
|
52
|
+
self.class.backends.each do |backend_class|
|
|
53
|
+
return backend_class.new.fetch(url)
|
|
54
|
+
rescue Ask::WebFetch::Error => e
|
|
55
|
+
failures << "#{backend_class.backend_name}: #{e.message}"
|
|
73
56
|
end
|
|
57
|
+
raise Ask::WebFetch::Error,
|
|
58
|
+
"all web fetch backends failed for #{url} (#{failures.join('; ')})"
|
|
74
59
|
end
|
|
75
60
|
|
|
76
|
-
def
|
|
77
|
-
doc = Nokogiri::HTML(html)
|
|
78
|
-
candidate = extract_main(doc)
|
|
79
|
-
scrub(candidate)
|
|
80
|
-
markdown = ReverseMarkdown.convert(candidate.to_html, unknown_tags: :bypass, github_flavored: true)
|
|
81
|
-
markdown = clean(markdown)
|
|
82
|
-
return "No readable content found at #{url}." if markdown.empty?
|
|
83
|
-
|
|
61
|
+
def format(page, url)
|
|
84
62
|
header = +''
|
|
85
|
-
title =
|
|
63
|
+
title = page[:title]
|
|
86
64
|
header << "# #{title}\n\n" unless title.to_s.empty?
|
|
87
65
|
header << "Source: #{url}\n\n"
|
|
88
|
-
header +
|
|
89
|
-
end
|
|
90
|
-
|
|
91
|
-
def extract_main(doc)
|
|
92
|
-
doc.at('article') || doc.at('main') || doc.at('[role="main"]') || doc.at('body') || doc
|
|
93
|
-
end
|
|
94
|
-
|
|
95
|
-
# Remove chrome INSIDE the candidate only, never ancestors.
|
|
96
|
-
def scrub(candidate)
|
|
97
|
-
candidate.css('script, style, noscript, nav, footer, header, iframe, form, svg, aside').each(&:remove)
|
|
98
|
-
candidate.css('*[id], *[class]').each do |el|
|
|
99
|
-
next if el.equal?(candidate)
|
|
100
|
-
|
|
101
|
-
id_cls = [el['id'], el['class']].compact.join(' ')
|
|
102
|
-
el.remove if id_cls.match?(NAV_CHROME_RE)
|
|
103
|
-
end
|
|
104
|
-
candidate
|
|
105
|
-
end
|
|
106
|
-
|
|
107
|
-
def clean(markdown)
|
|
108
|
-
markdown.gsub(/[ \t]+\n/, "\n")
|
|
109
|
-
.gsub(/\n{3,}/, "\n\n")
|
|
110
|
-
.strip
|
|
66
|
+
header + page[:content]
|
|
111
67
|
end
|
|
112
68
|
|
|
113
69
|
def truncate(markdown, max_chars)
|
data/lib/ask/web_fetch.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ask-web-fetch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kaka Ruto
|
|
@@ -107,10 +107,10 @@ dependencies:
|
|
|
107
107
|
- - "~>"
|
|
108
108
|
- !ruby/object:Gem::Version
|
|
109
109
|
version: '3.26'
|
|
110
|
-
description:
|
|
111
|
-
its content to clean markdown for LLM consumption.
|
|
112
|
-
+ reverse_markdown)
|
|
113
|
-
chat or agent.
|
|
110
|
+
description: Provides Ask::Tools::WebFetch, a tool that fetches a URL and converts
|
|
111
|
+
its content to clean markdown for LLM consumption. Defaults to a pure Ruby backend
|
|
112
|
+
(Net::HTTP + Nokogiri + reverse_markdown) with an automatic Jina Reader fallback
|
|
113
|
+
for JS-rendered or blocked pages. Works with any ask-rb chat or agent.
|
|
114
114
|
email:
|
|
115
115
|
- kaka@myrrlabs.com
|
|
116
116
|
executables: []
|
|
@@ -121,6 +121,9 @@ files:
|
|
|
121
121
|
- README.md
|
|
122
122
|
- lib/ask-web-fetch.rb
|
|
123
123
|
- lib/ask/web_fetch.rb
|
|
124
|
+
- lib/ask/web_fetch/backend.rb
|
|
125
|
+
- lib/ask/web_fetch/backends/jina.rb
|
|
126
|
+
- lib/ask/web_fetch/backends/local.rb
|
|
124
127
|
- lib/ask/web_fetch/tool.rb
|
|
125
128
|
- lib/ask/web_fetch/version.rb
|
|
126
129
|
homepage: https://github.com/ask-rb/ask-web-fetch
|