deepsearch-rb 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 435d04eb1b8c5a7d2d8e86cd17b863766559f82ffe4f194597e9b05a087b5ae0
+  data.tar.gz: 1f394f33996d8f85b17e17709a3178debb8ccad84d8268bd7590b532ac0d9fa7
+SHA512:
+  metadata.gz: 30634e66e50d377e755fe82354255d8b1e265578aca7e8f4c4833e8959ff33277a9a120c56b96c93efb13ea6e890cec8ddbf06d5952c0ef384efaceb3f8f8839
+  data.tar.gz: fca34ac34b09e4a8c5558c8114853dfec7c982fdf4bb9f630068c8ebf60bc398b1f711d5443b4e30aedc04c78539de24dcbf8a33761a719eb9815d8b38e37499

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog
+
+## [Released]
+
+## [0.1.0] - 2025-07-12
+
+### Added
+- Initial release

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Alexander Shagov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,138 @@
+# deepsearch-rb
+
+A ruby gem for performing LLM-powered automated web search.
+The only runtime dependencies are [ruby_llm](https://github.com/crmne/ruby_llm?tab=readme-ov-file) and [async](https://github.com/socketry/async).
+
+![single cycle diagram](docs/high-level-diagram.png)
+
+<details>
+<summary>🎥 A simple demo within a sinatra app demonstrating the basic 1-cycle flow. (`examples/simple_webapp`) ▶️</summary>
+
+![demo](docs/sinatra-app-demo.gif)
+</details>
+
+---
+
+**NOTE**: You can implement your own chains in the way it works for you, BFS/DFS search on any topic. A draft code might look like: 
+
+```
+Deepsearch.search(initial search) -> 
+  LLM(Generate additional queries) ->
+    Async [for each additional query]{ Deepsearch.search(sub-query) } ->
+      Aggregate()
+```
+
+## Installation
+
+[![Ruby](https://github.com/alexshagov/deepsearch-rb/actions/workflows/ruby.yml/badge.svg)](https://github.com/alexshagov/deepsearch-rb/actions/workflows/ruby.yml)
+
+### Quick installation via nix flakes
+
+If you're using nix flakes, just run `nix develop` and you'll enter a fully prepared dev environment.
+
+Otherwise, make sure you have installed `required_ruby_version` = ">= 3.2.0".
+
+```ruby
+# Gemfile
+gem 'deepsearch-rb'
+# ..
+
+bundle install
+```
+
+Or install from a local folder:
+```bash
+gem 'deepsearch-rb', path: '<path_to_gem>'
+```
+
+## Examples
+
+See the the `examples/` folder for a simple test as well as mini sinatra
+application streaming gem events via websocket interface. 
+
+## Configuration Options
+
+### Minimal config
+
+- The LLM configuration is fully based on [ruby_llm](https://github.com/crmne/ruby_llm?tab=readme-ov-file) gem.
+- There are two built-in search adapters you can use out of the box:
+  - [tavily](http://tavily.com/)
+  - [serper](http://serper.dev/playground)
+
+```ruby
+Deepsearch.configure do |config|
+  config.search_adapter = :tavily  # or :serper, :mock
+  config.tavily_api_key = "your_tavily_api_key"
+  config.serper_api_key = "your_serper_api_key"
+  
+  # LLM configuration (all config options are coming from `ruby_llm` gem),
+  # you can experiment with any model you like
+  config.ruby_llm.gemini_api_key = ENV['GEMINI_API_KEY']
+  config.ruby_llm.default_model = 'gemini-2.0-flash-lite'
+  config.ruby_llm.default_embedding_model = 'text-embedding-004'
+  # ..
+end
+```
+
+### Advanced config
+
+#### Specifying your own search adapter
+
+```ruby
+class MyCustomAdapter
+  def initializeт;end
+
+  def search(query, options = {})
+    # Implement your search logic here
+    {
+      "results" => [
+        { "url" => "https://example.com/result1", "content" => "Content 1 from custom search" }
+      ]
+    }
+  end
+end
+
+Deepsearch.configure { |config| config.custom_search_adapter_class = MyCustomAdapter }
+```
+
+#### Specifying your own prompts
+
+See the `Deepsearch::PromptsConfig` interface to learn what methods and arguments are expected.
+
+```ruby
+class MyPromptsConfig < Deepsearch::PromptsConfig
+  def subquery_prompt(query:)
+    <<~PROMPT
+      You are a search expert. Given this query: "#{query}",
+      generate 3 alternative search queries that would help find more information.
+      Return them as a simple list, one per line.
+    PROMPT
+  end
+
+  # Override other prompt methods as needed
+end
+
+Deepsearch.configure { |config| config.prompts_config = MyPromptsConfig.new }
+```
+
+#### Specifying event listener
+
+```ruby
+class MyListener
+  def on_deepsearch_event(event, step:, result:)
+    puts "Event: #{event}, Step: #{step}, Success: #{result.success?}"
+  end
+end
+
+Deepsearch.configure { |c| c.listener = MyListener.new }
+```
+
+#### Specifying max total search results
+
+```ruby
+result = Deepsearch.search("Ruby 3 unknown features", max_total_search_results: 25)
+```
+
+# License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/deepsearch/configuration.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+require_relative "prompts_config"
+
+module Deepsearch
+  # Encapsulates configuration options for the underlying `ruby_llm` gem.
+  # This provides a clean namespace for LLM settings within Deepsearch's configuration.
+  #
+  # @example
+  #   Deepsearch.configure do |config|
+  #     # Configure LLM settings via the `ruby_llm` accessor
+  #     config.ruby_llm.openai_api_key = "sk-..."
+  #     config.ruby_llm.default_model = "gpt-4o-mini"
+  #     config.ruby_llm.request_timeout = 90
+  #   end
+  class RubyLLMConfig
+    SUPPORTED_ATTRIBUTES = %i[
+      openai_api_key openai_organization_id openai_project_id
+      anthropic_api_key gemini_api_key deepseek_api_key openrouter_api_key
+      ollama_api_base bedrock_api_key bedrock_secret_key bedrock_region
+      bedrock_session_token openai_api_base default_model
+      default_embedding_model default_image_model request_timeout max_retries
+      retry_interval retry_backoff_factor retry_interval_randomness
+      http_proxy logger log_file log_level log_assume_model_exists
+    ].freeze
+
+    attr_accessor(*SUPPORTED_ATTRIBUTES)
+
+    def initialize
+      # Set some sensible defaults for Deepsearch's use case
+      @default_model = "gpt-4o-mini"
+      @default_embedding_model = "text-embedding-3-small"
+      @request_timeout = 30 # seconds
+      @log_assume_model_exists = false
+    end
+  end
+
+  # Configuration class for managing gem settings
+  class Configuration
+    # @!attribute listener
+    #   An object that can listen to events from the Deepsearch pipeline.
+    #   The object must respond to `on_deepsearch_event(event_name, **payload)`.
+    #   @example
+    #     class MyListener
+    #       def on_deepsearch_event(event, step:, result:)
+    #         puts "Event: #{event}, Step: #{step}, Success: #{result.success?}"
+    #       end
+    #     end
+    #     Deepsearch.configure { |c| c.listener = MyListener.new
+    attr_accessor :tavily_api_key, :serper_api_key, :search_adapter, :custom_search_adapter_class, :logger, :listener, :prompts
+    attr_reader :ruby_llm
+
+    def initialize
+      @tavily_api_key = nil
+      @serper_api_key = nil
+      @search_adapter = :tavily
+      @custom_search_adapter_class = nil
+      @listener = nil
+      @logger = Logger.new($stdout, level: Logger::DEBUG)
+      @ruby_llm = RubyLLMConfig.new
+      @prompts = PromptsConfig.new
+    end
+
+    # Reset configuration to default values
+    def reset!
+      @tavily_api_key = nil
+      @serper_api_key = nil
+      @search_adapter = :tavily
+      @listener = nil
+      @logger = Logger.new($stdout, level: Logger::DEBUG)
+      @ruby_llm = RubyLLMConfig.new
+      @prompts = PromptsConfig.new
+    end
+
+    # Configure RubyLLM with current settings from the `ruby_llm` config object.
+    def configure_llm!
+      require "ruby_llm"
+
+      RubyLLM.configure do |config|
+        RubyLLMConfig::SUPPORTED_ATTRIBUTES.each do |attr|
+          value = @ruby_llm.public_send(attr)
+          # Only set the value if it's not nil to avoid overriding RubyLLM's internal defaults.
+          config.public_send("#{attr}=", value) unless value.nil?
+        end
+      end
+    end
+  end
+end

data/lib/deepsearch/engine/pipeline.rb

@@ -0,0 +1,126 @@
+# frozen_string_literal: true
+
+require_relative "steps/prepare_subqueries/process"
+require_relative "steps/parallel_search/process"
+require_relative "steps/data_aggregation/process"
+require_relative "steps/rag/process"
+require_relative "steps/summarization/process"
+
+module Deepsearch
+  class Engine
+    # Orchestrates the entire multi-step search and summarization process.
+    # The pipeline executes a sequence of steps:
+    # 1. Prepares sub-queries from the initial query.
+    # 2. Performs parallel searches to gather website links.
+    # 3. Aggregates and parses content from the found websites.
+    # 4. Uses RAG to find text chunks relevant to the query.
+    # 5. Summarizes the relevant chunks into a final answer.
+    # It includes retry logic for each step to enhance robustness.
+    class Pipeline
+      def initialize(search_adapter)
+        @search_adapter = search_adapter
+      end
+
+      def execute(query, **options)
+        query_preprocessing_result = with_retry do
+          Steps::PrepareSubqueries::Process.new(query).execute
+        end
+        notify_listener(:step_completed, step: :prepare_subqueries, result: query_preprocessing_result)
+        # [query_preprocessing_result] Contains:
+        #   - cleaned_query [String] The sanitized version of original query
+        #   - original_query [String] The unmodified input query
+        #   - sub_queries [Array<String>] Generated subqueries (empty array on error)
+        #   - error [String, nil] Error message if processing failed
+        
+
+        parallel_search_options = {
+          initial_query: query_preprocessing_result.cleaned_query,
+          sub_queries: query_preprocessing_result.sub_queries,
+          search_adapter: @search_adapter,
+          **options
+        }
+
+        parallel_search_result = with_retry { Steps::ParallelSearch::Process.new(**parallel_search_options).execute }
+        notify_listener(:step_completed, step: :parallel_search, result: parallel_search_result)
+        # [parallel_search_result] Contains:        
+        #   - websites [Array<ParallelSearch::Result>] Search results
+        #     - ParallelSearch::Result objects with:
+        #       - websites [Array<Hash#url>] Array of website URLs
+        #   - success [Boolean] Whether search succeeded
+        #   - error [String, nil] Error message if search failed
+
+        data_aggregation_result = with_retry do
+          Steps::DataAggregation::Process.new(
+            websites: parallel_search_result.websites,
+          ).execute
+        end
+        notify_listener(:step_completed, step: :data_aggregation, result: data_aggregation_result)
+        # [data_aggregation_result] Contains:        
+        #   - parsed_websites [Array<DataAggregation::Result>]
+        #     - DataAggregation::Result objects with:
+        #       - url [String] Website URL
+        #       - content [String] Parsed content from the website
+        #   - success [Boolean] Whether search succeeded
+        #   - error [String, nil] Error message if search failed
+        
+        rag_result = with_retry do
+          Steps::Rag::Process.new(
+            query: query_preprocessing_result.cleaned_query,
+            parsed_websites: data_aggregation_result.parsed_websites
+          ).execute
+        end
+        notify_listener(:step_completed, step: :rag, result: rag_result)
+        # [rag_result] Contains:
+        #   - query [::Deepsearch::Engine::Steps::Rag::Values::Query]
+        #   - relevant_chunks [Array<::Deepsearch::Engine::Steps::Rag::Values::Chunk>]
+        summarization_result = with_retry do
+          Steps::Summarization::Process.new(
+            query: rag_result.query,
+            relevant_chunks: rag_result.relevant_chunks
+          ).execute
+        end
+        notify_listener(:step_completed, step: :summarization, result: summarization_result)
+        # [summarization_result] Contains:
+        #   - summary [String] The final answer with citations
+        #   - success [Boolean]
+        #   - error [String, nil]
+
+        summarization_result
+      end
+
+      private
+
+      def notify_listener(event, **payload)
+        listener = Deepsearch.configuration.listener
+        if !listener.respond_to?(:on_deepsearch_event)
+          Deepsearch.configuration.logger.debug("Attached listener does not respond to on_deepsearch_event, skipping notification")
+          return
+        end
+
+        begin
+          listener.on_deepsearch_event(event, **payload)
+        rescue StandardError => e
+          Deepsearch.configuration.logger.debug("Deepsearch listener failed: #{e.message}")
+        end
+      end
+
+      def with_retry(&block)
+        retries = 0
+        begin
+          result = block.call
+          # Handle "soft" failures from steps that return a result object with a #failure? method
+          raise "Operation failed: #{result.error}" if result.respond_to?(:failure?) && result.failure?
+      
+          result
+        rescue => e          
+          if (retries += 1) <= 1
+            Deepsearch.configuration.logger.debug("Retrying after error: #{e.message}")
+            retry
+          else
+            raise e
+          end
+        end
+      end
+    end
+  end
+end

data/lib/deepsearch/engine/steps/data_aggregation/parsed_website.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+require 'net/http'
+require 'uri'
+require 'nokogiri'
+
+module Deepsearch
+  class Engine
+    module Steps
+      module DataAggregation
+        # Fetches content from a URL, parses it, and cleans it to extract meaningful text.
+        # It handles HTTP requests, content type detection, and removal of unwanted HTML elements.
+        class ParsedWebsite
+          attr_reader :url, :content, :success, :error, :metadata, :timestamp
+
+          def initialize(url:)
+            @url = url
+            @content = nil
+            @success = false
+            @error = nil
+            fetch_content!
+          end
+
+          def success?
+            @success
+          end
+
+          def size
+            content.to_s.size
+          end
+
+          def to_h
+            {
+              url: url,
+              success: success?,
+              error: error,
+              content: content
+            }
+          end
+
+          protected
+
+          def fetch_content!
+            uri = URI.parse(@url)
+            
+            unless %w[http https].include?(uri.scheme)
+              @error = "Invalid URL scheme: #{uri.scheme}"
+              return
+            end
+
+            http = Net::HTTP.new(uri.host, uri.port)
+            http.use_ssl = uri.scheme == 'https'
+            http.read_timeout = 10
+            http.open_timeout = 5
+
+            request = Net::HTTP::Get.new(uri.request_uri)
+            request['User-Agent'] = random_user_agent
+
+            response = http.request(request)
+
+            if response.is_a?(Net::HTTPSuccess)
+              body = response.body.encode('UTF-8', invalid: :replace, undef: :replace, replace: '')
+              @content = clean_content(body)
+              @success = true
+            else
+              @error = "HTTP #{response.code}"
+            end
+          rescue => e
+            @error = e.message
+          end
+
+          private
+
+          def random_user_agent
+            [
+              'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 DeepSearch/1.0',
+              'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 DeepSearch/1.0',
+              'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15 DeepSearch/1.0',
+              'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0 DeepSearch/1.0'
+            ].sample
+          end
+
+          def clean_content(content)
+            raw_content = content.to_s
+            # If not HTML, we still need to make sure it's valid UTF-8 for JSON serialization.
+            unless raw_content =~ /<html[\s>]|<!DOCTYPE html/i
+              return raw_content.encode("UTF-8", "binary", invalid: :replace, undef: :replace, replace: "")
+            end
+
+            # Let Nokogiri parse the raw bytes and detect encoding
+            doc = Nokogiri::HTML(raw_content)
+
+            # Remove unwanted elements
+            doc.css('script, style, head, meta, link, noscript, iframe, svg, img').remove
+
+            # Remove comments
+            doc.xpath('//comment()').remove
+
+            # Remove inline styles and event handlers
+            doc.css('*').each do |node|
+              node.remove_attribute('style')
+              node.remove_attribute('onclick')
+              node.remove_attribute('onload')
+              node.remove_attribute('onerror')
+            end
+
+            # Get text content and clean it up
+            text = (doc.at('body')&.text || doc.text).to_s
+            utf8_text = text.encode("UTF-8", invalid: :replace, undef: :replace, replace: "")
+            utf8_text.gsub(/[[:space:]]+/, " ").strip
+          rescue StandardError
+            # Fallback if Nokogiri fails. The raw_content is the problem. Sanitize it from binary to UTF-8.
+            fallback_text = content.to_s.encode("UTF-8", "binary", invalid: :replace, undef: :replace, replace: "")
+            fallback_text.gsub(/<script\b[^>]*>.*?<\/script>/mi, "").gsub(/<style\b[^>]*>.*?<\/style>/mi, "").gsub(
+              /[[:space:]]+/, " "
+            ).strip
+          end
+        end
+      end
+    end
+  end
+end

data/lib/deepsearch/engine/steps/data_aggregation/process.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+require "async"
+require "async/semaphore"
+require_relative "parsed_website"
+require_relative "result"
+
+module Deepsearch
+  class Engine
+    module Steps
+      module DataAggregation
+        # Takes a list of website URLs from a previous search step and processes them in parallel.
+        # For each URL, it fetches, parses, and cleans the content using `ParsedWebsite`.
+        # It aggregates the successfully parsed websites into a `Result` object.
+        class Process
+          MAX_CONCURRENCY = 30
+
+          attr_reader :websites
+
+          def initialize(websites: [])
+            @websites = websites
+          end
+
+          def execute
+            Deepsearch.configuration.logger.debug("Starting data aggregation for #{@websites.size} websites")
+
+            parsed_websites = process_in_parallel
+            parsed_websites.filter!(&:success?)
+
+            Result.new(
+              parsed_websites: parsed_websites
+            )
+          rescue StandardError => e
+            Result.new(
+              parsed_websites: [],
+              error: e.message
+            )
+          end
+
+          private
+
+          def process_in_parallel
+            Sync do |task|
+              semaphore = Async::Semaphore.new(MAX_CONCURRENCY, parent: task)
+              websites.map do |website|
+                semaphore.async do
+                  ParsedWebsite.new(url: website['url'])
+                end
+              end.map(&:wait)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/deepsearch/engine/steps/data_aggregation/result.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Deepsearch
+  class Engine
+    module Steps
+      module DataAggregation
+        # Represents the result of the data aggregation step.
+        # It holds the collection of successfully parsed websites and any potential error message.
+        class Result
+          attr_reader :parsed_websites, :error
+
+          def initialize(parsed_websites: [], error: nil)
+            @parsed_websites = parsed_websites
+            @error = error
+          end
+
+          def success?
+            error.nil?
+          end
+
+          def failure?
+            !success?
+          end
+        end
+      end
+    end
+  end
+end

data/lib/deepsearch/engine/steps/parallel_search/process.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require_relative 'result'
+require_relative 'search'
+
+module Deepsearch
+  class Engine
+    module Steps
+      module ParallelSearch
+        # Orchestrates the parallel execution of multiple search queries (initial query + sub-queries).
+        # It uses the `Search` class to perform the actual concurrent searches via a search adapter
+        # and wraps the outcome in a `Result` object.
+        class Process
+          attr_reader :initial_query, :sub_queries, :search_adapter, :options
+
+          def initialize(initial_query:,
+            sub_queries:,
+            search_adapter:,
+            **options)
+            @initial_query = initial_query
+            @sub_queries = sub_queries
+            @search_adapter = search_adapter
+            @options = options
+          end
+
+          def execute
+            websites = Search.new(initial_query, sub_queries, search_adapter, **@options).output
+            Deepsearch.configuration.logger.debug("Parallel search completed with #{websites.size} results")
+            ParallelSearch::Result.new(
+              websites: websites
+            )
+          rescue StandardError => e
+            ParallelSearch::Result.new(
+              websites: [],
+              error: e.message
+            )
+          end
+        end
+      end
+    end
+  end
+end

data/lib/deepsearch/engine/steps/parallel_search/result.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Deepsearch
+  class Engine
+    module Steps
+      module ParallelSearch
+        # Represents the result of the parallel search step.
+        # It holds the aggregated list of websites found and any potential error message.
+        class Result
+          attr_reader :query, :websites, :error, :search_duration
+
+          def initialize(websites: [], error: nil)
+            @websites = websites || []
+            @error = error
+          end
+
+          def success?
+            error.nil?
+          end
+
+          def failure?
+            !success?
+          end
+        end
+      end
+    end
+  end
+end