ask-web-fetch 0.3.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +39 -9
- data/lib/ask/web_fetch/backends/crawl4ai.rb +4 -1
- data/lib/ask/web_fetch/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9286a544b4924f7dcde61b273e367240a81f8702c62510228e2d97f330c3617e
|
|
4
|
+
data.tar.gz: '08f55182a88cbc0c62e77f2acbc0570dc09cd28476f15caa78868b4e6519afd8'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c868cc6a828fea7d421064199991e054c57fb85d02a2d7713ea043d567960060da687a998095ee64e6ae0e6e1424a2219e7046b193077dfac7128e99256c485b
|
|
7
|
+
data.tar.gz: b071c880ba9594cd21778e4debff5894a36eb8d2fbddd7aabbf88efcf0195c922d7a1aa84b957ddcecdc4364aafa6c87328166ebe7e1d533f5b3d4e3dfacbd4d
|
data/README.md
CHANGED
|
@@ -12,32 +12,58 @@ required.
|
|
|
12
12
|
`Ask::Tools::WebFetch` runs a chain of pluggable backends and returns the
|
|
13
13
|
first success:
|
|
14
14
|
|
|
15
|
-
1. **
|
|
15
|
+
1. **Crawl4AI** (when configured) — self-hosted headless-Chromium renderer
|
|
16
|
+
(`POST /crawl` on `CRAWL4AI_URL`, default `http://localhost:11235`).
|
|
17
|
+
Renders JavaScript and returns clean fit-markdown, so it handles the
|
|
18
|
+
SPA pages the Local backend can't. Set `CRAWL4AI_URL` and it leads the
|
|
19
|
+
chain; when the service is down or unreachable it fails fast and falls
|
|
20
|
+
through.
|
|
21
|
+
2. **Local** (default) — pure Ruby `Net::HTTP` + Nokogiri + reverse_markdown:
|
|
16
22
|
browser-like User-Agent, redirects followed, main content extracted
|
|
17
23
|
(`<article>` → `<main>` → `<body>`), navigation/scripts stripped, tables
|
|
18
24
|
become markdown tables, links become `[text](url)`.
|
|
19
|
-
|
|
25
|
+
3. **Jina** — Jina Reader free tier (`https://r.jina.ai/<url>`). It runs
|
|
20
26
|
headless Chromium, so it renders JS pages the Local backend can't. Free
|
|
21
27
|
without a key (~20 req/min per IP); set `JINA_API_KEY` for higher limits.
|
|
22
28
|
|
|
23
|
-
The tool falls back automatically: if
|
|
24
|
-
non-HTML, anti-bot challenge, or a JS page with no
|
|
25
|
-
|
|
26
|
-
the call returns a failure result listing each
|
|
29
|
+
The tool falls back automatically: if Crawl4AI is absent or fails, Local is
|
|
30
|
+
tried (blocked, timeout, non-HTML, anti-bot challenge, or a JS page with no
|
|
31
|
+
server-side content), then Jina. If every backend fails (rate limit, access
|
|
32
|
+
error, challenge page), the call returns a failure result listing each
|
|
33
|
+
backend's error.
|
|
34
|
+
|
|
35
|
+
### Self-hosted Crawl4AI
|
|
36
|
+
|
|
37
|
+
[Crawl4AI](https://docs.crawl4ai.com) runs as its own Docker service — the
|
|
38
|
+
same self-hosted pattern as ask-web-search's SearXNG:
|
|
39
|
+
|
|
40
|
+
```sh
|
|
41
|
+
docker run -d --name crawl4ai -p 11235:11235 unclecode/crawl4ai:latest
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
```ruby
|
|
45
|
+
# lib/ask/web_fetch/backends/crawl4ai.rb is used automatically when:
|
|
46
|
+
ENV["CRAWL4AI_URL"] = "http://localhost:11235" # default when unset
|
|
47
|
+
ENV["CRAWL4AI_TOKEN"] = "..." # JWT-protected servers (0.9+)
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
When `CRAWL4AI_URL` is set the default chain is
|
|
51
|
+
`Crawl4Ai, Local, Jina`; otherwise it stays `Local, Jina`, so consumers
|
|
52
|
+
without a Crawl4AI service see no behavior change.
|
|
27
53
|
|
|
28
54
|
### Adding a backend
|
|
29
55
|
|
|
30
56
|
Backends subclass `Ask::WebFetch::Backend` and implement one method:
|
|
31
57
|
|
|
32
58
|
```ruby
|
|
33
|
-
class
|
|
59
|
+
class MyBackend < Ask::WebFetch::Backend
|
|
34
60
|
def fetch(url)
|
|
35
61
|
# return { title: "Page Title", content: "markdown..." }
|
|
36
62
|
# or raise Ask::WebFetch::FetchError / EmptyContentError
|
|
37
63
|
end
|
|
38
64
|
end
|
|
39
65
|
|
|
40
|
-
Ask::Tools::WebFetch.backends = [
|
|
66
|
+
Ask::Tools::WebFetch.backends = [MyBackend, Ask::WebFetch::Backends::Local]
|
|
41
67
|
```
|
|
42
68
|
|
|
43
69
|
`#fetch` must return `{ title: String|nil, content: String }` and raise
|
|
@@ -87,13 +113,17 @@ backend's error.
|
|
|
87
113
|
|
|
88
114
|
No configuration required for the default chain. Optional knobs:
|
|
89
115
|
|
|
116
|
+
- `CRAWL4AI_URL` — enables the self-hosted Crawl4AI backend and leads the
|
|
117
|
+
chain (default `http://localhost:11235` when set via the class accessor)
|
|
118
|
+
- `CRAWL4AI_TOKEN` — Bearer token for JWT-protected Crawl4AI servers (0.9+)
|
|
90
119
|
- `JINA_API_KEY` — enables the Jina fallback with higher rate limits
|
|
91
120
|
- `max_chars` parameter — caps output length (default 20000)
|
|
92
121
|
|
|
93
122
|
## Known limitations
|
|
94
123
|
|
|
95
124
|
- Pages rendered entirely client-side (JavaScript SPAs) may yield little or
|
|
96
|
-
no content —
|
|
125
|
+
no content unless Crawl4AI is configured — set `CRAWL4AI_URL` to handle
|
|
126
|
+
them with a self-hosted renderer.
|
|
97
127
|
- Some sites block non-browser requests regardless of User-Agent.
|
|
98
128
|
|
|
99
129
|
## Full documentation
|
|
@@ -22,7 +22,10 @@ module Ask
|
|
|
22
22
|
class Crawl4Ai < Backend
|
|
23
23
|
DEFAULT_URL = 'http://localhost:11235'
|
|
24
24
|
OPEN_TIMEOUT = 5
|
|
25
|
-
|
|
25
|
+
# Browser rendering (plus first-request pool warmup) is slow — the
|
|
26
|
+
# crawl itself gets crawler_config.timeout, so the HTTP read must
|
|
27
|
+
# allow that plus headroom, unlike the plain-HTML backends.
|
|
28
|
+
READ_TIMEOUT = 90
|
|
26
29
|
CRAWL_TIMEOUT = 60
|
|
27
30
|
|
|
28
31
|
class << self
|