ask-web-fetch 0.2.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +39 -9
- data/lib/ask/web_fetch/backends/crawl4ai.rb +114 -0
- data/lib/ask/web_fetch/tool.rb +18 -6
- data/lib/ask/web_fetch/version.rb +1 -1
- data/lib/ask/web_fetch.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9286a544b4924f7dcde61b273e367240a81f8702c62510228e2d97f330c3617e
|
|
4
|
+
data.tar.gz: '08f55182a88cbc0c62e77f2acbc0570dc09cd28476f15caa78868b4e6519afd8'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c868cc6a828fea7d421064199991e054c57fb85d02a2d7713ea043d567960060da687a998095ee64e6ae0e6e1424a2219e7046b193077dfac7128e99256c485b
|
|
7
|
+
data.tar.gz: b071c880ba9594cd21778e4debff5894a36eb8d2fbddd7aabbf88efcf0195c922d7a1aa84b957ddcecdc4364aafa6c87328166ebe7e1d533f5b3d4e3dfacbd4d
|
data/README.md
CHANGED
|
@@ -12,32 +12,58 @@ required.
|
|
|
12
12
|
`Ask::Tools::WebFetch` runs a chain of pluggable backends and returns the
|
|
13
13
|
first success:
|
|
14
14
|
|
|
15
|
-
1. **
|
|
15
|
+
1. **Crawl4AI** (when configured) — self-hosted headless-Chromium renderer
|
|
16
|
+
(`POST /crawl` on `CRAWL4AI_URL`, default `http://localhost:11235`).
|
|
17
|
+
Renders JavaScript and returns clean fit-markdown, so it handles the
|
|
18
|
+
SPA pages the Local backend can't. Set `CRAWL4AI_URL` and it leads the
|
|
19
|
+
chain; when the service is down or unreachable it fails fast and falls
|
|
20
|
+
through.
|
|
21
|
+
2. **Local** (default) — pure Ruby `Net::HTTP` + Nokogiri + reverse_markdown:
|
|
16
22
|
browser-like User-Agent, redirects followed, main content extracted
|
|
17
23
|
(`<article>` → `<main>` → `<body>`), navigation/scripts stripped, tables
|
|
18
24
|
become markdown tables, links become `[text](url)`.
|
|
19
|
-
|
|
25
|
+
3. **Jina** — Jina Reader free tier (`https://r.jina.ai/<url>`). It runs
|
|
20
26
|
headless Chromium, so it renders JS pages the Local backend can't. Free
|
|
21
27
|
without a key (~20 req/min per IP); set `JINA_API_KEY` for higher limits.
|
|
22
28
|
|
|
23
|
-
The tool falls back automatically: if
|
|
24
|
-
non-HTML, anti-bot challenge, or a JS page with no
|
|
25
|
-
|
|
26
|
-
the call returns a failure result listing each
|
|
29
|
+
The tool falls back automatically: if Crawl4AI is absent or fails, Local is
|
|
30
|
+
tried (blocked, timeout, non-HTML, anti-bot challenge, or a JS page with no
|
|
31
|
+
server-side content), then Jina. If every backend fails (rate limit, access
|
|
32
|
+
error, challenge page), the call returns a failure result listing each
|
|
33
|
+
backend's error.
|
|
34
|
+
|
|
35
|
+
### Self-hosted Crawl4AI
|
|
36
|
+
|
|
37
|
+
[Crawl4AI](https://docs.crawl4ai.com) runs as its own Docker service — the
|
|
38
|
+
same self-hosted pattern as ask-web-search's SearXNG:
|
|
39
|
+
|
|
40
|
+
```sh
|
|
41
|
+
docker run -d --name crawl4ai -p 11235:11235 unclecode/crawl4ai:latest
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
```ruby
|
|
45
|
+
# lib/ask/web_fetch/backends/crawl4ai.rb is used automatically when:
|
|
46
|
+
ENV["CRAWL4AI_URL"] = "http://localhost:11235" # default when unset
|
|
47
|
+
ENV["CRAWL4AI_TOKEN"] = "..." # JWT-protected servers (0.9+)
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
When `CRAWL4AI_URL` is set the default chain is
|
|
51
|
+
`Crawl4Ai, Local, Jina`; otherwise it stays `Local, Jina`, so consumers
|
|
52
|
+
without a Crawl4AI service see no behavior change.
|
|
27
53
|
|
|
28
54
|
### Adding a backend
|
|
29
55
|
|
|
30
56
|
Backends subclass `Ask::WebFetch::Backend` and implement one method:
|
|
31
57
|
|
|
32
58
|
```ruby
|
|
33
|
-
class
|
|
59
|
+
class MyBackend < Ask::WebFetch::Backend
|
|
34
60
|
def fetch(url)
|
|
35
61
|
# return { title: "Page Title", content: "markdown..." }
|
|
36
62
|
# or raise Ask::WebFetch::FetchError / EmptyContentError
|
|
37
63
|
end
|
|
38
64
|
end
|
|
39
65
|
|
|
40
|
-
Ask::Tools::WebFetch.backends = [
|
|
66
|
+
Ask::Tools::WebFetch.backends = [MyBackend, Ask::WebFetch::Backends::Local]
|
|
41
67
|
```
|
|
42
68
|
|
|
43
69
|
`#fetch` must return `{ title: String|nil, content: String }` and raise
|
|
@@ -87,13 +113,17 @@ backend's error.
|
|
|
87
113
|
|
|
88
114
|
No configuration required for the default chain. Optional knobs:
|
|
89
115
|
|
|
116
|
+
- `CRAWL4AI_URL` — enables the self-hosted Crawl4AI backend and leads the
|
|
117
|
+
chain (default `http://localhost:11235` when set via the class accessor)
|
|
118
|
+
- `CRAWL4AI_TOKEN` — Bearer token for JWT-protected Crawl4AI servers (0.9+)
|
|
90
119
|
- `JINA_API_KEY` — enables the Jina fallback with higher rate limits
|
|
91
120
|
- `max_chars` parameter — caps output length (default 20000)
|
|
92
121
|
|
|
93
122
|
## Known limitations
|
|
94
123
|
|
|
95
124
|
- Pages rendered entirely client-side (JavaScript SPAs) may yield little or
|
|
96
|
-
no content —
|
|
125
|
+
no content unless Crawl4AI is configured — set `CRAWL4AI_URL` to handle
|
|
126
|
+
them with a self-hosted renderer.
|
|
97
127
|
- Some sites block non-browser requests regardless of User-Agent.
|
|
98
128
|
|
|
99
129
|
## Full documentation
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'net/http'
|
|
4
|
+
require 'uri'
|
|
5
|
+
require 'json'
|
|
6
|
+
require_relative '../backend'
|
|
7
|
+
|
|
8
|
+
module Ask
|
|
9
|
+
module WebFetch
|
|
10
|
+
module Backends
|
|
11
|
+
# Self-hosted Crawl4AI (https://docs.crawl4ai.com) — a headless
|
|
12
|
+
# Chromium crawler that renders JavaScript and returns clean markdown.
|
|
13
|
+
# Runs as its own Docker service (default http://localhost:11235), the
|
|
14
|
+
# same self-hosted pattern as ask-web-search's SearXNG. No API key;
|
|
15
|
+
# configure via CRAWL4AI_URL (and CRAWL4AI_TOKEN for 0.9+ JWT-protected
|
|
16
|
+
# servers).
|
|
17
|
+
#
|
|
18
|
+
# Kept FIRST in the default chain: when the service is present it
|
|
19
|
+
# handles the JS-rendered pages the Local backend can't. When it isn't
|
|
20
|
+
# configured — or is unreachable — it fails fast and the chain falls
|
|
21
|
+
# through to Local, with Jina as the last resort.
|
|
22
|
+
class Crawl4Ai < Backend
|
|
23
|
+
DEFAULT_URL = 'http://localhost:11235'
|
|
24
|
+
OPEN_TIMEOUT = 5
|
|
25
|
+
# Browser rendering (plus first-request pool warmup) is slow — the
|
|
26
|
+
# crawl itself gets crawler_config.timeout, so the HTTP read must
|
|
27
|
+
# allow that plus headroom, unlike the plain-HTML backends.
|
|
28
|
+
READ_TIMEOUT = 90
|
|
29
|
+
CRAWL_TIMEOUT = 60
|
|
30
|
+
|
|
31
|
+
class << self
|
|
32
|
+
attr_writer :url, :token
|
|
33
|
+
|
|
34
|
+
def url
|
|
35
|
+
@url || ENV['CRAWL4AI_URL']
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def token
|
|
39
|
+
@token || ENV['CRAWL4AI_TOKEN']
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# Presence = configuration. The tool's default chain only includes
|
|
43
|
+
# this backend when CRAWL4AI_URL is set, so consumers without a
|
|
44
|
+
# Crawl4AI service see zero behavior change (Local -> Jina).
|
|
45
|
+
def configured?
|
|
46
|
+
!url.to_s.empty?
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def fetch(url)
|
|
51
|
+
raise FetchError, 'Crawl4AI not configured (set CRAWL4AI_URL)' if self.class.url.to_s.empty?
|
|
52
|
+
|
|
53
|
+
body = crawl(url)
|
|
54
|
+
raise FetchError, "challenge page at #{url}" if challenge_page?(body)
|
|
55
|
+
|
|
56
|
+
page = to_page(body, url)
|
|
57
|
+
raise EmptyContentError, "no readable content at #{url}" unless usable_content?(page[:content])
|
|
58
|
+
|
|
59
|
+
page
|
|
60
|
+
rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
|
|
61
|
+
Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
|
|
62
|
+
raise FetchError, "Crawl4AI #{e.class}: #{e.message}"
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
private
|
|
66
|
+
|
|
67
|
+
def crawl(url)
|
|
68
|
+
uri = URI("#{self.class.url.chomp('/')}/crawl")
|
|
69
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
70
|
+
http.use_ssl = uri.scheme == 'https'
|
|
71
|
+
http.open_timeout = OPEN_TIMEOUT
|
|
72
|
+
http.read_timeout = READ_TIMEOUT
|
|
73
|
+
|
|
74
|
+
req = Net::HTTP::Post.new(uri)
|
|
75
|
+
req['Content-Type'] = 'application/json'
|
|
76
|
+
req['Accept'] = 'application/json'
|
|
77
|
+
req['User-Agent'] = USER_AGENT
|
|
78
|
+
req['Authorization'] = "Bearer #{self.class.token}" if self.class.token
|
|
79
|
+
req.body = JSON.generate(
|
|
80
|
+
urls: [url],
|
|
81
|
+
crawler_config: { cache_mode: 'bypass', timeout: CRAWL_TIMEOUT }
|
|
82
|
+
)
|
|
83
|
+
|
|
84
|
+
res = http.request(req)
|
|
85
|
+
case res.code
|
|
86
|
+
when '200'
|
|
87
|
+
res.body.to_s
|
|
88
|
+
when '401', '403'
|
|
89
|
+
raise FetchError, "Crawl4AI auth error (#{res.code})"
|
|
90
|
+
else
|
|
91
|
+
raise FetchError, "Crawl4AI returned #{res.code}"
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
# The /crawl response is {success:, results: [CrawlResult...]} where
|
|
96
|
+
# each result carries markdown (fit_markdown preferred, raw_markdown
|
|
97
|
+
# fallback) and metadata (title, description, ...).
|
|
98
|
+
def to_page(body, url)
|
|
99
|
+
data = JSON.parse(body)
|
|
100
|
+
result = Array(data['results']).first || {}
|
|
101
|
+
if result['success'] == false
|
|
102
|
+
raise FetchError, "Crawl4AI crawl failed: #{result['error_message'] || 'unknown error'}"
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
markdown = result.dig('markdown', 'fit_markdown').to_s
|
|
106
|
+
markdown = result.dig('markdown', 'raw_markdown').to_s if markdown.strip.empty?
|
|
107
|
+
{ title: result.dig('metadata', 'title'), content: markdown }
|
|
108
|
+
rescue JSON::ParserError => e
|
|
109
|
+
raise FetchError, "Crawl4AI bad JSON response: #{e.message}"
|
|
110
|
+
end
|
|
111
|
+
end
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
end
|
data/lib/ask/web_fetch/tool.rb
CHANGED
|
@@ -3,22 +3,34 @@
|
|
|
3
3
|
require 'ask-tools'
|
|
4
4
|
require_relative '../web_fetch/backend'
|
|
5
5
|
require_relative '../web_fetch/backends/local'
|
|
6
|
+
require_relative '../web_fetch/backends/crawl4ai'
|
|
6
7
|
require_relative '../web_fetch/backends/jina'
|
|
7
8
|
|
|
8
9
|
module Ask
|
|
9
10
|
module Tools
|
|
10
11
|
# Fetches a URL and returns its content as clean markdown for LLM
|
|
11
12
|
# consumption. Tries each configured backend in order and returns the
|
|
12
|
-
# first success
|
|
13
|
-
#
|
|
13
|
+
# first success.
|
|
14
|
+
#
|
|
15
|
+
# Chain: Crawl4AI first (self-hosted headless-Chromium renderer; used
|
|
16
|
+
# when the CRAWL4AI_URL service is present, fails fast when it isn't),
|
|
17
|
+
# then the local fetcher, with Jina Reader as the last resort.
|
|
14
18
|
class WebFetch < Ask::Tool
|
|
15
19
|
DEFAULT_MAX_CHARS = 20_000
|
|
16
20
|
|
|
17
|
-
# Backend chain, tried in order.
|
|
18
|
-
# (
|
|
19
|
-
#
|
|
21
|
+
# Backend chain, tried in order. Crawl4AI leads when configured
|
|
22
|
+
# (CRAWL4AI_URL), so a present self-hosted renderer is preferred;
|
|
23
|
+
# otherwise Local, with Jina as the last resort. Swap or extend for
|
|
24
|
+
# future backends; each must subclass Ask::WebFetch::Backend and
|
|
25
|
+
# implement #fetch(url).
|
|
20
26
|
def self.backends
|
|
21
|
-
@backends ||=
|
|
27
|
+
@backends ||= begin
|
|
28
|
+
chain = [Ask::WebFetch::Backends::Local, Ask::WebFetch::Backends::Jina]
|
|
29
|
+
if Ask::WebFetch::Backends::Crawl4Ai.configured?
|
|
30
|
+
chain.unshift(Ask::WebFetch::Backends::Crawl4Ai)
|
|
31
|
+
end
|
|
32
|
+
chain
|
|
33
|
+
end
|
|
22
34
|
end
|
|
23
35
|
|
|
24
36
|
class << self
|
data/lib/ask/web_fetch.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ask-web-fetch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kaka Ruto
|
|
@@ -122,6 +122,7 @@ files:
|
|
|
122
122
|
- lib/ask-web-fetch.rb
|
|
123
123
|
- lib/ask/web_fetch.rb
|
|
124
124
|
- lib/ask/web_fetch/backend.rb
|
|
125
|
+
- lib/ask/web_fetch/backends/crawl4ai.rb
|
|
125
126
|
- lib/ask/web_fetch/backends/jina.rb
|
|
126
127
|
- lib/ask/web_fetch/backends/local.rb
|
|
127
128
|
- lib/ask/web_fetch/tool.rb
|