RubyGems - ask-web-fetch - Versions diffs - 0.1.0 → 0.3.0 - Mend

ask-web-fetch 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (10) hide show

checksums.yaml +4 -4
data/README.md +45 -14
data/lib/ask/web_fetch/backend.rb +66 -0
data/lib/ask/web_fetch/backends/crawl4ai.rb +111 -0
data/lib/ask/web_fetch/backends/jina.rb +53 -0
data/lib/ask/web_fetch/backends/local.rb +101 -0
data/lib/ask/web_fetch/tool.rb +42 -74
data/lib/ask/web_fetch/version.rb +1 -1
data/lib/ask/web_fetch.rb +4 -0
metadata +9 -5

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e750ba3efd9f7947a21e35e92ae5e99d639bc028dd8c2be2a8581707a0fdd908
-  data.tar.gz: 5c8d428a526da1e0cc7ac1397adbb73e22eb4714c3576cf2b6ec927bbe5630f8
+  metadata.gz: efd68d22906c21155fe37405f68ef05185732ff6f1ce39767e50c87e4d535994
+  data.tar.gz: 7d06e49acbc527f06badb034933d792203910637c0e2671f133ec2fc159be41d
 SHA512:
-  metadata.gz: f770807fc25707aae50934aa6ebe4b2137fea07cda5e4204f9cf34ab441bf3f426a526c61c92edc467e011c32b712d00bf5c948cefafc110bb2f7c9139177725
-  data.tar.gz: f6d780234851e1cd40d96ab2f3f75636054e2cff4a13f6591d50caa7c621d09b833021831244e3df90eb4869facae6d50fa664fd8f53386f653a4692b00c1eac
+  metadata.gz: 7736807de8ff9c7b1bad298af12abf67af77f254090532f39f080d0dd98c2bb3a7fed60aac5cf8bbeaeb977b03d3d38900525a4a85b012dc94ae171f18f38b3e
+  data.tar.gz: 2edae586d7baeb9114e29d23540ccbaa331e920a8fe83b7acea12c612b8491116fc635c948d437a157c3193b90d63b97e84857b7714f73ff5f6d19d6c907472b

data/README.md CHANGED Viewed

@@ -4,19 +4,47 @@
 A web fetch tool for the ask-rb ecosystem. It provides
 `Ask::Tools::WebFetch`, which fetches a URL and returns its content as clean
-markdown for LLM consumption. It has no Rails dependencies, no external
-service, and no API key required — pure Ruby (`Net::HTTP` + Nokogiri +
-reverse_markdown).
+markdown for LLM consumption. It has no Rails dependencies and no API key
+required.
 ## How it works
-1. `Net::HTTP` GET with a browser-like User-Agent (redirects followed, up to 5)
-2. Nokogiri parses the HTML and picks the main content
-   (`<article>` → `<main>` → `<body>`)
-3. Navigation, scripts, and other chrome are stripped
-4. reverse_markdown converts the content to markdown (tables become markdown
-   tables, links become `[text](url)`)
-5. Output is truncated to `max_chars` (default 20000)
+`Ask::Tools::WebFetch` runs a chain of pluggable backends and returns the
+first success:
+1. **Local** (default) — pure Ruby `Net::HTTP` + Nokogiri + reverse_markdown:
+   browser-like User-Agent, redirects followed, main content extracted
+   (`<article>` → `<main>` → `<body>`), navigation/scripts stripped, tables
+   become markdown tables, links become `[text](url)`.
+2. **Jina** — Jina Reader free tier (`https://r.jina.ai/<url>`). It runs
+   headless Chromium, so it renders JS pages the Local backend can't. Free
+   without a key (~20 req/min per IP); set `JINA_API_KEY` for higher limits.
+The tool falls back automatically: if Local fails (blocked, timeout,
+non-HTML, anti-bot challenge, or a JS page with no server-side content), it
+tries Jina. If Jina fails too (rate limit, access error, challenge page),
+the call returns a failure result listing each backend's error.
+### Adding a backend
+Backends subclass `Ask::WebFetch::Backend` and implement one method:
+```ruby
+class Crawl4ai < Ask::WebFetch::Backend
+  def fetch(url)
+    # return { title: "Page Title", content: "markdown..." }
+    # or raise Ask::WebFetch::FetchError / EmptyContentError
+  end
+end
+Ask::Tools::WebFetch.backends = [Crawl4ai, Ask::WebFetch::Backends::Local]
+```
+`#fetch` must return `{ title: String|nil, content: String }` and raise
+`Ask::WebFetch::FetchError` (hard failure) or `EmptyContentError` (page
+fetched but nothing usable). The chain then handles ordering and fallback
+for you. For tests, `Ask::Tools::WebFetch.backends = [...]` can be swapped
+and restored.
 ## Installation
@@ -51,13 +79,16 @@ You can cap the output size:
 result = tool.execute(url: "https://example.com", max_chars: 5000)
 ```
-If the page has no readable content, returns
-`"No readable content found at <url>."`
+If every backend fails (e.g. the page is behind an anti-bot challenge and
+Jina is also rate-limited), the call returns a failure result with each
+backend's error.
 ## Configuration
-No configuration required. The only knob is the optional `max_chars`
-parameter.
+No configuration required for the default chain. Optional knobs:
+- `JINA_API_KEY` — enables the Jina fallback with higher rate limits
+- `max_chars` parameter — caps output length (default 20000)
 ## Known limitations

data/lib/ask/web_fetch/backend.rb ADDED Viewed

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+require_relative 'version'
+module Ask
+  module WebFetch
+    # Raised by backends on any failure; the tool catches it and tries the
+    # next backend in the chain.
+    class Error < StandardError; end
+    # A backend that failed to fetch (network error, non-2xx, challenge
+    # page, non-HTML response).
+    class FetchError < Error; end
+    # A backend that fetched the page but found nothing usable in it.
+    class EmptyContentError < Error; end
+    # Base class for fetch backends, plus the errors they raise.
+    #
+    # A backend turns a URL into LLM-ready markdown. To add a new backend:
+    #
+    #   1. subclass Backend and implement #fetch(url)
+    #   2. #fetch must return { title: String|nil, content: String }
+    #   3. #fetch must raise FetchError (hard failure) or
+    #      EmptyContentError (page fetched but nothing usable) on failure
+    #   4. register the class in Ask::Tools::WebFetch.backends
+    #
+    # The tool tries each backend in order and returns the first success.
+    class Backend
+      # Identity sent on every request, browser-like plus a gem tag.
+      USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
+                   'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 ' \
+                   "ask-web-fetch/#{Ask::WebFetch::VERSION}".freeze
+      # Content shorter than this is treated as a page with no usable
+      # content (e.g. a JS-rendered shell with nothing server-side).
+      MIN_CONTENT_LENGTH = 100
+      # Cloudflare-style anti-bot signatures. Deliberately narrow:
+      # challenge/interstitial pages carry these markers, while legitimate
+      # pages can contain the word "captcha" in unrelated config/JS (e.g.
+      # Wikipedia embeds an hcaptcha edit-config flag on every page).
+      CHALLENGE_RE = /just a moment|checking your browser|cf-chl/i
+      def self.backend_name
+        name.split('::').last
+      end
+      # Fetches +url+ and returns { title: String|nil, content: String }.
+      # Raises FetchError or EmptyContentError on failure.
+      def fetch(url)
+        raise NotImplementedError, "#{self.class} must implement #fetch(url)"
+      end
+      private
+      def challenge_page?(body)
+        body.to_s.match?(CHALLENGE_RE)
+      end
+      def usable_content?(content)
+        content.to_s.strip.length >= MIN_CONTENT_LENGTH
+      end
+    end
+  end
+end

data/lib/ask/web_fetch/backends/crawl4ai.rb ADDED Viewed

@@ -0,0 +1,111 @@
+# frozen_string_literal: true
+require 'net/http'
+require 'uri'
+require 'json'
+require_relative '../backend'
+module Ask
+  module WebFetch
+    module Backends
+      # Self-hosted Crawl4AI (https://docs.crawl4ai.com) — a headless
+      # Chromium crawler that renders JavaScript and returns clean markdown.
+      # Runs as its own Docker service (default http://localhost:11235), the
+      # same self-hosted pattern as ask-web-search's SearXNG. No API key;
+      # configure via CRAWL4AI_URL (and CRAWL4AI_TOKEN for 0.9+ JWT-protected
+      # servers).
+      #
+      # Kept FIRST in the default chain: when the service is present it
+      # handles the JS-rendered pages the Local backend can't. When it isn't
+      # configured — or is unreachable — it fails fast and the chain falls
+      # through to Local, with Jina as the last resort.
+      class Crawl4Ai < Backend
+        DEFAULT_URL = 'http://localhost:11235'
+        OPEN_TIMEOUT = 5
+        READ_TIMEOUT = 30
+        CRAWL_TIMEOUT = 60
+        class << self
+          attr_writer :url, :token
+          def url
+            @url || ENV['CRAWL4AI_URL']
+          end
+          def token
+            @token || ENV['CRAWL4AI_TOKEN']
+          end
+          # Presence = configuration. The tool's default chain only includes
+          # this backend when CRAWL4AI_URL is set, so consumers without a
+          # Crawl4AI service see zero behavior change (Local -> Jina).
+          def configured?
+            !url.to_s.empty?
+          end
+        end
+        def fetch(url)
+          raise FetchError, 'Crawl4AI not configured (set CRAWL4AI_URL)' if self.class.url.to_s.empty?
+          body = crawl(url)
+          raise FetchError, "challenge page at #{url}" if challenge_page?(body)
+          page = to_page(body, url)
+          raise EmptyContentError, "no readable content at #{url}" unless usable_content?(page[:content])
+          page
+        rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
+               Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
+          raise FetchError, "Crawl4AI #{e.class}: #{e.message}"
+        end
+        private
+        def crawl(url)
+          uri = URI("#{self.class.url.chomp('/')}/crawl")
+          http = Net::HTTP.new(uri.host, uri.port)
+          http.use_ssl = uri.scheme == 'https'
+          http.open_timeout = OPEN_TIMEOUT
+          http.read_timeout = READ_TIMEOUT
+          req = Net::HTTP::Post.new(uri)
+          req['Content-Type'] = 'application/json'
+          req['Accept'] = 'application/json'
+          req['User-Agent'] = USER_AGENT
+          req['Authorization'] = "Bearer #{self.class.token}" if self.class.token
+          req.body = JSON.generate(
+            urls: [url],
+            crawler_config: { cache_mode: 'bypass', timeout: CRAWL_TIMEOUT }
+          )
+          res = http.request(req)
+          case res.code
+          when '200'
+            res.body.to_s
+          when '401', '403'
+            raise FetchError, "Crawl4AI auth error (#{res.code})"
+          else
+            raise FetchError, "Crawl4AI returned #{res.code}"
+          end
+        end
+        # The /crawl response is {success:, results: [CrawlResult...]} where
+        # each result carries markdown (fit_markdown preferred, raw_markdown
+        # fallback) and metadata (title, description, ...).
+        def to_page(body, url)
+          data = JSON.parse(body)
+          result = Array(data['results']).first || {}
+          if result['success'] == false
+            raise FetchError, "Crawl4AI crawl failed: #{result['error_message'] || 'unknown error'}"
+          end
+          markdown = result.dig('markdown', 'fit_markdown').to_s
+          markdown = result.dig('markdown', 'raw_markdown').to_s if markdown.strip.empty?
+          { title: result.dig('metadata', 'title'), content: markdown }
+        rescue JSON::ParserError => e
+          raise FetchError, "Crawl4AI bad JSON response: #{e.message}"
+        end
+      end
+    end
+  end
+end

data/lib/ask/web_fetch/backends/jina.rb ADDED Viewed

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+require 'net/http'
+require 'uri'
+require_relative '../backend'
+module Ask
+  module WebFetch
+    module Backends
+      # Jina Reader free tier: GET https://r.jina.ai/<url>.
+      #
+      # Free without a key (~20 req/min per IP). Set JINA_API_KEY for higher
+      # rate limits. The endpoint runs headless Chromium, so it renders JS
+      # pages that the Local backend cannot.
+      class Jina < Backend
+        BASE_URL = 'https://r.jina.ai'
+        OPEN_TIMEOUT = 5
+        READ_TIMEOUT = 30
+        def fetch(url)
+          uri = URI("#{BASE_URL}/#{url}")
+          http = Net::HTTP.new(uri.host, uri.port)
+          http.use_ssl = true
+          http.open_timeout = OPEN_TIMEOUT
+          http.read_timeout = READ_TIMEOUT
+          req = Net::HTTP::Get.new(uri)
+          req['User-Agent'] = USER_AGENT
+          req['Accept'] = 'text/markdown, text/plain, text/html'
+          req['Authorization'] = "Bearer #{ENV['JINA_API_KEY']}" if ENV['JINA_API_KEY']
+          res = http.request(req)
+          case res.code
+          when '200'
+            body = res.body.to_s
+            raise FetchError, 'challenge page from Jina' if challenge_page?(body)
+            raise EmptyContentError, 'empty response from Jina' unless usable_content?(body)
+            { title: nil, content: body.strip }
+          when '429'
+            raise FetchError, 'rate limited by Jina (429)'
+          when '401', '403'
+            raise FetchError, "Jina access error (#{res.code})"
+          else
+            raise FetchError, "Jina returned #{res.code}"
+          end
+        rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
+               Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
+          raise FetchError, "Jina #{e.class}: #{e.message}"
+        end
+      end
+    end
+  end
+end

data/lib/ask/web_fetch/backends/local.rb ADDED Viewed

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+require 'net/http'
+require 'uri'
+require 'nokogiri'
+require 'reverse_markdown'
+require_relative '../backend'
+module Ask
+  module WebFetch
+    module Backends
+      # Default backend: pure Ruby Net::HTTP + Nokogiri + reverse_markdown.
+      # No external service or API key; mirrors ask-web-search's self-hosted
+      # SearXNG approach.
+      class Local < Backend
+        MAX_REDIRECTS = 5
+        OPEN_TIMEOUT = 5
+        READ_TIMEOUT = 15
+        # Class/id fragments that mark navigation chrome worth dropping, e.g.
+        # "vector-page-toolbar", "sidebar", "toc".
+        NAV_CHROME_RE = /
+          (^|[\s_-])(nav|menu|toolbar|breadcrumb|sidebar|toc|footer|header|
+          banner|pagination|search|cookie|modal|popup)([\s_-]|$)
+        /ix
+        def fetch(url)
+          body, content_type = fetch_html(url)
+          raise FetchError, "expected HTML from #{url}, got #{content_type}" unless content_type.include?('html')
+          raise FetchError, "challenge page at #{url}" if challenge_page?(body)
+          page = to_markdown(body, url)
+          raise EmptyContentError, "no readable content at #{url}" unless usable_content?(page[:content])
+          page
+        rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
+               Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
+          raise FetchError, "#{e.class}: #{e.message}"
+        end
+        # Parses +html+ and returns { title:, content: } where content is
+        # clean markdown.
+        def to_markdown(html, _url)
+          doc = Nokogiri::HTML(html)
+          candidate = extract_main(doc)
+          scrub(candidate)
+          markdown = ReverseMarkdown.convert(candidate.to_html, unknown_tags: :bypass, github_flavored: true)
+          markdown = clean(markdown)
+          title = doc.at('title')&.text&.strip
+          { title: title, content: markdown }
+        end
+        private
+        # GET with redirect following (max MAX_REDIRECTS hops).
+        def fetch_html(url)
+          uri = URI(url)
+          hops = 0
+          loop do
+            http = Net::HTTP.new(uri.host, uri.port)
+            http.open_timeout = OPEN_TIMEOUT
+            http.read_timeout = READ_TIMEOUT
+            http.use_ssl = uri.scheme == 'https'
+            req = Net::HTTP::Get.new(uri)
+            req['User-Agent'] = USER_AGENT
+            req['Accept'] = 'text/html,application/xhtml+xml'
+            res = http.request(req)
+            return [res.body, res['content-type'].to_s] if res.code.start_with?('2')
+            raise FetchError, "got #{res.code} from #{url}" unless res.code.start_with?('3') && res['location']
+            raise FetchError, "hit a redirect loop at #{url}" if (hops += 1) > MAX_REDIRECTS
+            uri = URI.join(uri, res['location'])
+          end
+        end
+        def extract_main(doc)
+          doc.at('article') || doc.at('main') || doc.at('[role="main"]') || doc.at('body') || doc
+        end
+        # Remove chrome INSIDE the candidate only, never ancestors.
+        def scrub(candidate)
+          candidate.css('script, style, noscript, nav, footer, header, iframe, form, svg, aside').each(&:remove)
+          candidate.css('*[id], *[class]').each do |el|
+            next if el.equal?(candidate)
+            id_cls = [el['id'], el['class']].compact.join(' ')
+            el.remove if id_cls.match?(NAV_CHROME_RE)
+          end
+          candidate
+        end
+        def clean(markdown)
+          markdown.gsub(/[ \t]+\n/, "\n")
+                  .gsub(/\n{3,}/, "\n\n")
+                  .strip
+        end
+      end
+    end
+  end
+end

data/lib/ask/web_fetch/tool.rb CHANGED Viewed

@@ -1,31 +1,41 @@
 # frozen_string_literal: true
 require 'ask-tools'
-require 'net/http'
-require 'uri'
-require 'nokogiri'
-require 'reverse_markdown'
+require_relative '../web_fetch/backend'
+require_relative '../web_fetch/backends/local'
+require_relative '../web_fetch/backends/crawl4ai'
+require_relative '../web_fetch/backends/jina'
 module Ask
   module Tools
     # Fetches a URL and returns its content as clean markdown for LLM
-    # consumption. Pure Ruby: Net::HTTP + Nokogiri + reverse_markdown.
-    # No external service or API key required.
+    # consumption. Tries each configured backend in order and returns the
+    # first success.
+    #
+    # Chain: Crawl4AI first (self-hosted headless-Chromium renderer; used
+    # when the CRAWL4AI_URL service is present, fails fast when it isn't),
+    # then the local fetcher, with Jina Reader as the last resort.
     class WebFetch < Ask::Tool
-      USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
-                   'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 ' \
-                   "ask-web-fetch/#{Ask::WebFetch::VERSION}".freeze
       DEFAULT_MAX_CHARS = 20_000
-      MAX_REDIRECTS = 5
-      OPEN_TIMEOUT = 5
-      READ_TIMEOUT = 15
-      # Class/id fragments that mark navigation chrome worth dropping, e.g.
-      # "vector-page-toolbar", "sidebar", "toc".
-      NAV_CHROME_RE = /
-        (^|[\s_-])(nav|menu|toolbar|breadcrumb|sidebar|toc|footer|header|
-        banner|pagination|search|cookie|modal|popup)([\s_-]|$)
-      /ix
+      # Backend chain, tried in order. Crawl4AI leads when configured
+      # (CRAWL4AI_URL), so a present self-hosted renderer is preferred;
+      # otherwise Local, with Jina as the last resort. Swap or extend for
+      # future backends; each must subclass Ask::WebFetch::Backend and
+      # implement #fetch(url).
+      def self.backends
+        @backends ||= begin
+          chain = [Ask::WebFetch::Backends::Local, Ask::WebFetch::Backends::Jina]
+          if Ask::WebFetch::Backends::Crawl4Ai.configured?
+            chain.unshift(Ask::WebFetch::Backends::Crawl4Ai)
+          end
+          chain
+        end
+      end
+      class << self
+        attr_writer :backends
+      end
       description 'Fetch a URL and return its content as clean markdown for LLM consumption. ' \
                   'Use this to read web pages, articles, and documentation.'
@@ -40,74 +50,32 @@ module Ask
       )
       def execute(url:, max_chars: DEFAULT_MAX_CHARS)
-        body, content_type = fetch(url)
-        raise "WebFetch expected HTML from #{url}, got #{content_type}" unless content_type.include?('html')
-        markdown = to_markdown(body, url)
+        page = fetch_page(url)
+        markdown = format(page, url)
         markdown = truncate(markdown, max_chars) if max_chars&.positive?
         Ask::Result.ok(data: markdown)
       end
       private
-      # GET with redirect following (max MAX_REDIRECTS hops). Raises on
-      # non-2xx responses; Ask::Tool#call wraps exceptions into a failure.
-      def fetch(url)
-        uri = URI(url)
-        hops = 0
-        loop do
-          http = Net::HTTP.new(uri.host, uri.port)
-          http.open_timeout = OPEN_TIMEOUT
-          http.read_timeout = READ_TIMEOUT
-          http.use_ssl = uri.scheme == 'https'
-          req = Net::HTTP::Get.new(uri)
-          req['User-Agent'] = USER_AGENT
-          req['Accept'] = 'text/html,application/xhtml+xml'
-          res = http.request(req)
-          return [res.body, res['content-type'].to_s] if res.code.start_with?('2')
-          raise "WebFetch got #{res.code} from #{url}" unless res.code.start_with?('3') && res['location']
-          raise "WebFetch hit a redirect loop at #{url}" if (hops += 1) > MAX_REDIRECTS
-          uri = URI.join(uri, res['location'])
+      # Try each configured backend in order; return the first success.
+      def fetch_page(url)
+        failures = []
+        self.class.backends.each do |backend_class|
+          return backend_class.new.fetch(url)
+        rescue Ask::WebFetch::Error => e
+          failures << "#{backend_class.backend_name}: #{e.message}"
         end
+        raise Ask::WebFetch::Error,
+              "all web fetch backends failed for #{url} (#{failures.join('; ')})"
       end
-      def to_markdown(html, url)
-        doc = Nokogiri::HTML(html)
-        candidate = extract_main(doc)
-        scrub(candidate)
-        markdown = ReverseMarkdown.convert(candidate.to_html, unknown_tags: :bypass, github_flavored: true)
-        markdown = clean(markdown)
-        return "No readable content found at #{url}." if markdown.empty?
+      def format(page, url)
         header = +''
-        title = doc.at('title')&.text&.strip
+        title = page[:title]
         header << "# #{title}\n\n" unless title.to_s.empty?
         header << "Source: #{url}\n\n"
-        header + markdown
-      end
-      def extract_main(doc)
-        doc.at('article') || doc.at('main') || doc.at('[role="main"]') || doc.at('body') || doc
-      end
-      # Remove chrome INSIDE the candidate only, never ancestors.
-      def scrub(candidate)
-        candidate.css('script, style, noscript, nav, footer, header, iframe, form, svg, aside').each(&:remove)
-        candidate.css('*[id], *[class]').each do |el|
-          next if el.equal?(candidate)
-          id_cls = [el['id'], el['class']].compact.join(' ')
-          el.remove if id_cls.match?(NAV_CHROME_RE)
-        end
-        candidate
-      end
-      def clean(markdown)
-        markdown.gsub(/[ \t]+\n/, "\n")
-                .gsub(/\n{3,}/, "\n\n")
-                .strip
+        header + page[:content]
       end
       def truncate(markdown, max_chars)

data/lib/ask/web_fetch/version.rb CHANGED Viewed

@@ -2,6 +2,6 @@
 module Ask
   module WebFetch
-    VERSION = '0.1.0'
+    VERSION = '0.3.0'
   end
 end

data/lib/ask/web_fetch.rb CHANGED Viewed

@@ -1,4 +1,8 @@
 # frozen_string_literal: true
 require_relative 'web_fetch/version'
+require_relative 'web_fetch/backend'
+require_relative 'web_fetch/backends/local'
+require_relative 'web_fetch/backends/crawl4ai'
+require_relative 'web_fetch/backends/jina'
 require_relative 'web_fetch/tool'

metadata CHANGED Viewed

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ask-web-fetch
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Kaka Ruto
@@ -107,10 +107,10 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.26'
-description: 'Provides Ask::Tools::WebFetch, a tool that fetches a URL and converts
-  its content to clean markdown for LLM consumption. Pure Ruby (Net::HTTP + Nokogiri
-  + reverse_markdown): no external service or API key required. Works with any ask-rb
-  chat or agent.'
+description: Provides Ask::Tools::WebFetch, a tool that fetches a URL and converts
+  its content to clean markdown for LLM consumption. Defaults to a pure Ruby backend
+  (Net::HTTP + Nokogiri + reverse_markdown) with an automatic Jina Reader fallback
+  for JS-rendered or blocked pages. Works with any ask-rb chat or agent.
 email:
 - kaka@myrrlabs.com
 executables: []
@@ -121,6 +121,10 @@ files:
 - README.md
 - lib/ask-web-fetch.rb
 - lib/ask/web_fetch.rb
+- lib/ask/web_fetch/backend.rb
+- lib/ask/web_fetch/backends/crawl4ai.rb
+- lib/ask/web_fetch/backends/jina.rb
+- lib/ask/web_fetch/backends/local.rb
 - lib/ask/web_fetch/tool.rb
 - lib/ask/web_fetch/version.rb
 homepage: https://github.com/ask-rb/ask-web-fetch