scraper_utils 0.8.2 → 0.8.3

data/docs/fibers_and_threads.md

@@ -1,72 +0,0 @@
-Fibers and Threads
-==================
-
-This sequence diagram supplements the notes on the {ScraperUtils::Scheduler} class and is intended to help show
-the passing of messages and control between the fibers and threads.
-
-* To keep things simple I have only shown the Fibers and Threads and not all the other calls like to the
-  OperationRegistry to lookup the current operation, or OperationWorker etc.
-* There is ONE (global) response queue, which is monitored by the Scheduler.run_operations loop in the main Fiber
-* Each authority has ONE OperationWorker (not shown), which has ONE Fiber, ONE Thread, ONE request queue.
-* I use "◀─▶" to indicate a call and response, and "║" for which fiber / object is currently running.
-
-```text
-
-SCHEDULER (Main Fiber)      
-NxRegister-operation  RESPONSE.Q   
-   ║──creates────────◀─▶┐
-   ║                    │       FIBER (runs block passed to register_operation)
-   ║──creates──────────────────◀─▶┐         WORKER object and Registry
-   ║──registers(fiber)───────────────────────▶┐           REQUEST-Q
-   │                    │         │           ║──creates─◀─▶┐       THREAD
-   │                    │         │           ║──creates───────────◀─▶┐
-   ║◀─────────────────────────────────────────┘             ║◀──pop───║ ...[block waiting for request]
-   ║                    │         │           │             ║         │
-run_operations          │         │           │             ║         │
-   ║──pop(non block)─◀─▶│         │           │             ║         │ ...[no responses yet]
-   ║                    │         │           │             ║         │
-   ║───resumes-next─"can_resume"─────────────▶┐             ║         │
-   │                    │         │           ║             ║         │
-   │                    │         ║◀──resume──┘             ║         │ ...[first Resume passes true]
-   │                    │         ║           │             ║         │ ...[initialise scraper]
-```
-**REPEATS FROM HERE**  
-```text
- SCHEDULER        RESPONSE.Q    FIBER       WORKER        REQUEST.Q THREAD
-   │                    │         ║──request─▶┐             ║         │
-   │                    │         │           ║──push req ─▶║         │
-   │                    │         ║◀──────────┘             ║──req───▶║
-   ║◀──yields control─(waiting)───┘           │             │         ║ 
-   ║                    │         │           │             │         ║ ...[Executes network I/O request]
-   ║                    │         │           │             │         ║
-   ║───other-resumes... │         │           │             │         ║ ...[Other Workers will be resumed 
-   ║                    │         │           │             │         ║     till most 99% are waiting on 
-   ║───lots of          │         │           │             │         ║     responses from their threads
-   ║    short sleeps    ║◀──pushes response───────────────────────────┘
-   ║                    ║         │           │             ║◀──pop───║ ...[block waiting for request]
-   ║──pop(response)──◀─▶║         │           │             ║         │
-   ║                    │         │           │             ║         │
-   ║──saves─response───────────────────────◀─▶│             ║         │
-   ║                    │         │           │             ║         │
-   ║───resumes-next─"can_resume"─────────────▶┐             ║         │
-   │                    │         │           ║             ║         │
-   │                    │         ║◀──resume──┘             ║         │ ...[Resume passes response]
-   │                    │         ║           │             ║         │ 
-   │                    │         ║           │             ║         │ ...[Process Response]
-```
-**REPEATS TO HERE** - WHEN FIBER FINISHES, instead it:  
-```text
- SCHEDULER        RESPONSE.Q    FIBER         WORKER        REQUEST.Q THREAD  
-   │                    │         ║             │           ║         │ 
-   │                    │         ║─deregister─▶║           ║         │
-   │                    │         │             ║──close───▶║         │ 
-   │                    │         │             ║           ║──nil───▶┐ 
-   │                    │         │             ║           │         ║ ... [thread exists] 
-   │                    │         │             ║──join────────────◀─▶┘ 
-   │                    │         │             ║  ....... [worker removes 
-   │                    │         │             ║           itself from registry]                     
-   │                    │         ║◀──returns───┘                       
-   │◀──returns─nil────────────────┘                                    
-   │                    │                                             
-```
-When the last fiber finishes and the registry is empty, then the response queue is also removed

data/docs/interleaving_requests.md

@@ -1,33 +0,0 @@
-# Interleaving Requests with Scheduler
-
-The `ScraperUtils::Scheduler` provides a lightweight utility that:
-
-* Works on other authorities while in the delay period for an authority's next request
-* Optimizes the total scraper run time
-* Allows you to increase the random delay for authorities without undue effect on total run time
-* For the curious, it uses [ruby fibers](https://ruby-doc.org/core-2.5.8/Fiber.html) rather than threads as that is
-  a simpler system and thus easier to get right, understand and debug!
-* Cycles around the authorities when compliant_mode, max_load and random_delay are disabled
-
-## Implementation
-
-To enable fiber scheduling, change your scrape method as per
-{example_scrape_with_fibers.rb example scrape with fibers}
-
-## Logging with Scheduler
-
-Use {ScraperUtils::LogUtils.log} instead of `puts` when logging within the authority processing code.
-This will prefix the output lines with the authority name, which is needed since the system will interleave the work and
-thus the output.
-
-## Testing Considerations
-
-This uses {ScraperUtils::RandomizeUtils} for determining the order of operations. Remember to add the following line to
-`spec/spec_helper.rb`:
-
-```ruby
-ScraperUtils::RandomizeUtils.random = false
-ScraperUtils::Scheduler.max_workers = 1
-```
-
-For full details, see the {Scheduler}.

data/docs/parallel_requests.md

@@ -1,138 +0,0 @@
-Parallel Request Processing
-===========================
-
-The ScraperUtils library provides a mechanism for executing network I/O requests in parallel using a thread for each
-operation worker, allowing the fiber to yield control and allow other fibers to process whilst the thread processes the
-mechanize network I/O request.
-
-This can be disabled by setting `MORPH_DISABLE_THREADS` ENV var to a non-blank value.
-
-Overview
---------
-
-When scraping multiple authority websites, around 99% of the time was spent waiting for network I/O. While the
-`Scheduler`
-efficiently interleaves fibers during delay periods, network I/O requests will still block a fiber until they
-complete.
-
-The `OperationWorker` optimizes this process by:
-
-1. Executing mechanize network operations in parallel using a thread for each operation_worker and fiber
-2. Allowing other fibers to continue working while waiting for thread responses
-3. Integrating seamlessly with the existing `Scheduler`
-
-Usage
------
-
-```ruby
-# In your authority scraper block
-ScraperUtils::Scheduler.register_operation("authority_name") do
-  # Instead of:
-  # page = agent.get(url)
-
-  # Use:
-  page = ScraperUtils::Scheduler.execute_request(agent, :get, [url])
-
-  # Process page as normal
-  process_page(page)
-end
-```
-
-For testing purposes, you can also execute non-network operations:
-
-```ruby
-# Create a test object
-test_object = Object.new
-
-def test_object.sleep_test(duration)
-  sleep(duration)
-  "Completed after #{duration} seconds"
-end
-
-# Queue a sleep command
-command = ScraperUtils::ProcessRequest.new(
-  "test_id",
-  test_object,
-  :sleep_test,
-  [0.5]
-)
-
-thread_scheduler.queue_request(command)
-```
-
-Configuration
--------------
-
-The followingENV variables affect how `Scheduler` is configured:
-
-* `MORPH_DISABLE_THREADS=1` disabled the use of threads
-* `MORPH_MAX_WORKERS=N` configures the system to a max of N workers (minimum 1).
-  If N is 1 then this forces the system to process one authority at a time.
-
-Key Components
---------------
-
-### ThreadRequest
-
-A value object encapsulating a command to be executed:
-
-- External ID: Any value suitable as a hash key (String, Symbol, Integer, Object) that identifies the command
-- Subject: The object to call the method on
-- Method: The method to call on the subject
-- Args: Arguments to pass to the method
-
-### ThreadResponse
-
-A value object encapsulating a response:
-
-- External ID: Matches the ID from the original command
-- Result: The result of the operation
-- Error: Any error that occurred
-- Time Taken: Execution time in seconds
-
-### ThreadPool
-
-Manages a pool of threads that execute commands:
-
-- Processes commands from a queue
-- Returns responses with matching external IDs
-- Provides clear separation between I/O and scheduling
-
-Benefits
---------
-
-1. **Improved Throughput**: Process multiple operations simultaneously
-2. **Reduced Total Runtime**: Make better use of wait time during network operations
-3. **Optimal Resource Usage**: Efficiently balance CPU and network operations
-4. **Better Geolocation Handling**: Distribute requests across proxies more efficiently
-5. **Testability**: Execute non-network operations for testing concurrency
-
-Debugging
----------
-
-When debugging issues with parallel operations, use:
-
-```shell
-# Set debug level to see request/response logging
-export DEBUG = 2
-```
-
-The system will log:
-
-- When commands are queued
-- When responses are received
-- How long each operation took
-- Any errors that occurred
-
-## Implementation Details
-
-The integration between `Scheduler` and `ThreadPool` follows these principles:
-
-1. `Scheduler` maintains ownership of all fiber scheduling
-2. `ThreadPool` only knows about commands and responses
-3. Communication happens via value objects with validation
-4. State is managed in dedicated `FiberState` objects
-5. Each component has a single responsibility
-
-This design provides a clean separation of concerns while enabling parallel operations within the existing fiber
-scheduling framework.

data/docs/randomizing_requests.md

@@ -1,38 +0,0 @@
-Randomizing Requests
-====================
-
-`ScraperUtils::RandomizeUtils` provides utilities for randomizing processing order in scrapers,
-which is helpful for distributing load and avoiding predictable patterns.
-
-Usage
------
-
-Pass a `Collection` or `Array` to `ScraperUtils::RandomizeUtils.randomize_order` to randomize it in production, but
-receive it as is when testing.
-
-```ruby
-# Randomize a collection
-randomized_authorities = ScraperUtils::RandomizeUtils.randomize_order(authorities)
-
-# Use with a list of records from an index to randomize requests for details
-records.each do |record|
-  # Process record
-end
-```
-
-Testing Configuration
----------------------
-
-Enforce sequential mode when testing by adding the following code to `spec/spec_helper.rb`:
-
-```ruby
-ScraperUtils::RandomizeUtils.random = false
-```
-
-Notes
------
-
-* You can also disable random mode by setting the env variable `MORPH_DISABLE_RANDOM` to `1` (or any non-blank value)
-* Testing using VCR requires random to be disabled
-
-For full details, see {ScraperUtils::RandomizeUtils Randomize Utils class documentation}

data/docs/reducing_server_load.md

@@ -1,63 +0,0 @@
-# Reducing Server Load
-
-This document explains various techniques for reducing load on the servers you're scraping.
-
-## Intelligent Date Range Selection
-
-To further reduce server load and speed up scrapers, we provide an intelligent date range selection mechanism
-that can reduce server requests by 60% without significantly impacting delay in picking up changes.
-
-The `ScraperUtils::DateRangeUtils#calculate_date_ranges` method provides a smart approach to searching historical
-records:
-
-- Always checks the most recent 4 days daily (configurable)
-- Progressively reduces search frequency for older records
-- Uses a progression from each 2 days and upwards to create an efficient search intervals
-- Configurable `max_period` (default is 2 days)
-- Merges adjacent search ranges and handles the changeover in search frequency by extending some searches
-
-Example usage in your scraper:
-
-```ruby
-date_ranges = ScraperUtils::DateRangeUtils.new.calculate_date_ranges
-date_ranges.each do |from_date, to_date, _debugging_comment|
-  # Adjust your normal search code to use for this date range
-  your_search_records(from_date: from_date, to_date: to_date) do |record|
-    # process as normal
-  end
-end
-```
-
-Typical server load compared to search all days each time:
-
-* Max period 2 days : ~59% of the 33 days selected (default, alternates between 57% and 61% covered)
-* Max period 3 days : ~50% of the 33 days selected (varies much more - between 33 and 67%)
-* Max period 4 days : ~46% (more efficient if you search back 50 or more days, varies between 15 and 61%)
-
-See the [DateRangeUtils class documentation](https://rubydoc.info/gems/scraper_utils/ScraperUtils/DateRangeUtils) for customizing defaults and passing options.
-
-## Cycle Utilities
-
-Simple utility for cycling through options based on Julian day number to reduce server load and make your scraper seem less bot-like.
-
-If the site uses tags like 'L28', 'L14' and 'L7' for the last 28, 14 and 7 days, an alternative solution
-is to cycle through ['L28', 'L7', 'L14', 'L7'] which would drop the load by 50% and be less bot-like.
-
-```ruby
-# Toggle between main and alternate behaviour
-alternate = ScraperUtils::CycleUtils.position(2).even?
-
-# OR cycle through a list of values day by day:
-period = ScraperUtils::CycleUtils.pick(['L28', 'L7', 'L14', 'L7'])
-
-# Use with any cycle size
-pos = ScraperUtils::CycleUtils.position(7) # 0-6 cycle
-
-# Test with specific date
-pos = ScraperUtils::CycleUtils.position(3, date: Date.new(2024, 1, 5))
-
-# Override for testing
-# CYCLE_POSITION=2 bundle exec ruby scraper.rb
-```
-
-For full details, see the [CycleUtils class documentation](https://rubydoc.info/gems/scraper_utils/ScraperUtils/CycleUtils).

data/lib/scraper_utils/cycle_utils.rb

@@ -1,26 +0,0 @@
-# frozen_string_literal: true
-
-module ScraperUtils
-  # Provides utilities for cycling through a range of options day by day
-  module CycleUtils
-    # Returns position in cycle from zero onwards
-    # @param cycle [Integer] Cycle size (2 onwards)
-    # @param date [Date, nil] Optional date to use instead of today
-    # @return [Integer] position in cycle progressing from zero to cycle-1 and then repeating day by day
-    # Can override using CYCLE_POSITION ENV variable
-    def self.position(cycle, date: nil)
-      day = ENV.fetch('CYCLE_POSITION', (date || Date.today).jd).to_i
-      day % cycle
-    end
-
-    # Returns one value per day, cycling through all possible values in order
-    # @param values [Array] Values to cycle through
-    # @param date [Date, nil] Optional date to use instead of today to calculate position
-    # @return value from array
-    # Can override using CYCLE_POSITION ENV variable
-    def self.pick(values, date: nil)
-      values = values.to_a
-      values[position(values.size, date: date)]
-    end
-  end
-end

data/lib/scraper_utils/date_range_utils.rb

@@ -1,118 +0,0 @@
-# frozen_string_literal: true
-
-module ScraperUtils
-  class DateRangeUtils
-    MERGE_ADJACENT_RANGES = true
-    PERIODS = [2, 3, 4].freeze
-
-    class << self
-      # @return [Integer] Default number of days to cover
-      attr_accessor :default_days
-
-      # @return [Integer] Default days to always include in ranges
-      attr_accessor :default_everytime
-
-      # @return [Integer, nil] Default max days between any one date being in a range
-      attr_accessor :default_max_period
-
-      # Configure default settings for all DateRangeUtils instances
-      # @yield [self] Yields self for configuration
-      # @example
-      #   AgentConfig.configure do |config|
-      #     config.default_everytime = 3
-      #     config.default_days = 35
-      #     config.default_max_period = 5
-      #   end
-      # @return [void]
-      def configure
-        yield self if block_given?
-      end
-
-      # Reset all configuration options to their default values
-      # @return [void]
-      def reset_defaults!
-        @default_days = ENV.fetch('MORPH_DAYS', 33).to_i # 33
-        @default_everytime = ENV.fetch('MORPH_EVERYTIME', 4).to_i # 4
-        @default_max_period = ENV.fetch('MORPH_MAX_PERIOD', 2).to_i # 3
-      end
-    end
-
-    # Set defaults on load
-    reset_defaults!
-
-    attr_reader :max_period_used
-    attr_reader :extended_max_period
-
-    # Generates one or more date ranges to check the most recent daily through to checking each max_period
-    # There is a graduated schedule from the latest `everytime` days through to the oldest of `days` dates which is checked each `max_period` days.
-    # @param days [Integer, nil] create ranges that cover the last `days` dates
-    # @param everytime [Integer, nil] Always include the latest `everytime` out of `days` dates (minimum 1)
-    # @param max_period [Integer, nil] the last `days` dates must be checked at least every `max_period` days (1..4)
-    # @param today [Date, nil] overrides the default determination of today at UTC+09:30 (middle of Australia)
-    # @return [Array{[Date, Date, String]}] being from_date, to_date and a comment
-    #
-    # Uses a Fibonacci sequence to create a natural progression of check frequencies.
-    # Newer data is checked more frequently, with periods between checks growing
-    # according to the Fibonacci sequence (2, 3, 5, 8, 13...) until reaching max_period.
-    # This creates an efficient schedule that mimics natural information decay patterns.
-    def calculate_date_ranges(days: nil, everytime: nil, max_period: nil, today: nil)
-      _calculate_date_ranges(
-        Integer(days || self.class.default_days),
-        [1, Integer(everytime || self.class.default_everytime)].max,
-        Integer(max_period || self.class.default_max_period),
-        today || Time.now(in: '+09:30').to_date
-      )
-    end
-
-    private
-
-    def _calculate_date_ranges(days, everytime, max_period, today)
-      @max_period_used = 1
-      to_date = today
-      valid_periods = PERIODS.select { |p| p <= max_period }
-      if !max_period.positive? || !days.positive?
-        return []
-      elsif valid_periods.empty? || everytime >= days
-        # cover everything everytime
-        return [[today + 1 - days, today, "everything"]]
-      end
-      max_period = valid_periods.max
-      @max_period_used = max_period
-
-      one_half = ((days - everytime) / 2).to_i
-      one_third = ((days - everytime) / 3).to_i
-      two_ninths = (2 * (days - everytime) / 9).to_i
-      run_ranges =
-        case max_period
-        when 2
-          [
-            [[to_date - (one_half + everytime), to_date, "#{max_period}#0+everytime"]],
-            [[to_date - days, to_date - (one_half + everytime), "#{max_period}#1"], [to_date - everytime, to_date, "everytime"]]
-          ]
-        when 3
-          [
-            [[to_date - days - 1, to_date + two_ninths - days, "3#0"], [to_date - (one_third + everytime), to_date, "2#0+everytime"]],
-            [[to_date + two_ninths - days, to_date + 2 * two_ninths - days, "3#1"], [to_date - everytime, to_date, "everytime"]],
-            [[to_date + 2 * two_ninths - days, to_date, "3#2+2#0+everytime"]],
-            [[to_date - days - 1, to_date + two_ninths - days, "3#3"], [to_date - everytime, to_date, "everytime"]],
-            [[to_date + two_ninths - days, to_date + 2 * two_ninths - days, "3#4"], [to_date - (one_third + everytime), to_date, "2#2+everytime"]],
-            [[to_date + 2 * two_ninths - days, to_date - (one_third + everytime), "3#5"], [to_date - everytime, to_date, "everytime"]]
-          ]
-        else
-          [
-            [[to_date - (one_half + everytime), to_date, "2#0+everytime"]],
-            [[to_date - days - 2, to_date - (one_half + everytime), "4#0"], [to_date - everytime, to_date, "everytime"]],
-            [[to_date - (one_half + everytime), to_date, "2#1+everytime"]],
-            [[to_date - everytime, to_date, "everytime"]]
-          ]
-        end
-      run_number = today.to_date.jd % run_ranges.size
-
-      ranges = run_ranges[run_number]
-      if days.positive? && ScraperUtils::DebugUtils.trace?
-        LogUtils.log "DEBUG: #{max_period} ranges: #{ranges.inspect}"
-      end
-      ranges
-    end
-  end
-end

data/lib/scraper_utils/mechanize_actions.rb

@@ -1,183 +0,0 @@
-# frozen_string_literal: true
-
-module ScraperUtils
-  # Class for executing a series of mechanize actions with flexible replacements
-  #
-  # @example Basic usage
-  #   agent = ScraperUtils::MechanizeUtils.mechanize_agent
-  #   page = agent.get("https://example.com")
-  #   
-  #   actions = [
-  #     [:click, "Next Page"],
-  #     [:click, ["Option A", "xpath://div[@id='results']/a", "css:.some-button"]] # Will select one randomly
-  #   ]
-  #   
-  #   processor = ScraperUtils::MechanizeActions.new(agent)
-  #   result_page = processor.process(page, actions)
-  #
-  # @example With replacements
-  #   replacements = { FROM_DATE: "2022-01-01", TO_DATE: "2022-03-01" }
-  #   processor = ScraperUtils::MechanizeActions.new(agent, replacements)
-  #   
-  #   # Use replacements in actions
-  #   actions = [
-  #     [:click, "Search between {FROM_DATE} and {TO_DATE}"]
-  #   ]
-  class MechanizeActions
-    # @return [Mechanize] The mechanize agent used for actions
-    attr_reader :agent
-
-    # @return [Array] The results of each action performed
-    attr_reader :results
-
-    # Initialize a new MechanizeActions processor
-    #
-    # @param agent [Mechanize] The mechanize agent to use for actions
-    # @param replacements [Hash] Optional text replacements to apply to action parameters
-    def initialize(agent, replacements = {})
-      @agent = agent
-      @replacements = replacements || {}
-      @results = []
-    end
-
-    # Process a sequence of actions on a page
-    #
-    # @param page [Mechanize::Page] The starting page
-    # @param actions [Array<Array>] The sequence of actions to perform
-    # @return [Mechanize::Page] The resulting page after all actions
-    # @raise [ArgumentError] If an unknown action type is provided
-    #
-    # @example Action format
-    #   actions = [
-    #     [:click, "Link Text"],                     # Click on link with this text
-    #     [:click, ["Option A", "text:Option B"]],   # Click on one of these options (randomly selected)
-    #     [:click, "css:.some-button"],              # Use CSS selector
-    #     [:click, "xpath://div[@id='results']/a"],  # Use XPath selector
-    #     [:block, ->(page, args, agent, results) { [page, { custom_results: 'data' }] }] # Custom block
-    #   ]
-    def process(page, actions)
-      @results = []
-      current_page = page
-
-      actions.each do |action|
-        args = action.dup
-        action_type = args.shift
-        current_page, result =
-          case action_type
-          when :click
-            handle_click(current_page, args)
-          when :block
-            handle_block(current_page, args)
-          else
-            raise ArgumentError, "Unknown action type: #{action_type}"
-          end
-
-        @results << result
-      end
-
-      current_page
-    end
-
-    private
-
-    # Process a block action
-    #
-    # @param page [Mechanize::Page] The current page
-    # @param args [Array] The block and its arguments
-    # @return [Array<Mechanize::Page, Hash>] The resulting page and status
-    def handle_block(page, args)
-      block = args.shift
-      # Apply replacements to all remaining arguments
-      processed_args = args.map { |arg| apply_replacements(arg) }
-      block.call(page, processed_args.first, agent, @results.dup)
-    end
-
-    # Handle a click action
-    #
-    # @param page [Mechanize::Page] The current page
-    # @param args [Array] The first element is the selection target
-    # @return [Array<Mechanize::Page, Hash>] The resulting page and status
-    def handle_click(page, args)
-      target = args.shift
-      if target.is_a?(Array)
-        target = ScraperUtils::CycleUtils.pick(target, date: @replacements[:TODAY])
-      end
-      target = apply_replacements(target)
-      element = select_element(page, target)
-      if element.nil?
-        raise "Unable to find click target: #{target}"
-      end
-
-      result = { action: :click, target: target }
-      next_page = element.click
-      [next_page, result]
-    end
-
-    # Select an element on the page based on selector string
-    #
-    # @param page [Mechanize::Page] The page to search in
-    # @param selector_string [String] The selector string, optionally with "css:", "xpath:" or "text:" prefix
-    # @return [Mechanize::Element, nil] The selected element or nil if not found
-    def select_element(page, selector_string)
-      # Handle different selector types based on prefixes
-      if selector_string.start_with?("css:")
-        selector = selector_string.sub(/^css:/, '')
-        # We need to convert Nokogiri elements to Mechanize elements for clicking
-        css_element = page.at_css(selector)
-        return nil unless css_element
-        
-        # If it's a link, find the matching Mechanize link
-        if css_element.name.downcase == 'a' && css_element['href']
-          return page.links.find { |link| link.href == css_element['href'] }
-        end
-        
-        return css_element
-      elsif selector_string.start_with?("xpath:")
-        selector = selector_string.sub(/^xpath:/, '')
-        # We need to convert Nokogiri elements to Mechanize elements for clicking
-        xpath_element = page.at_xpath(selector)
-        return nil unless xpath_element
-        
-        # If it's a link, find the matching Mechanize link
-        if xpath_element.name.downcase == 'a' && xpath_element['href']
-          return page.links.find { |link| link.href == xpath_element['href'] }
-        end
-        
-        return xpath_element
-      else
-        # Default to text: for links
-        selector = selector_string.sub(/^text:/, '')
-        # Find links that include the text and don't have fragment-only hrefs
-        matching_links = page.links.select do |l|
-          l.text.include?(selector) &&
-            !(l.href.nil? || l.href.start_with?('#'))
-        end
-
-        if matching_links.empty?
-          # try case-insensitive
-          selector = selector.downcase
-          matching_links = page.links.select do |l|
-            l.text.downcase.include?(selector) &&
-              !(l.href.nil? || l.href.start_with?('#'))
-          end
-        end
-
-        # Get the link with the a. shortest (closest matching) text and then b. the longest href
-        matching_links.min_by { |l| [l.text.strip.length, -l.href.length] }
-      end
-    end
-
-    # Apply text replacements to a string
-    #
-    # @param text [String, Object] The text to process or object to return unchanged
-    # @return [String, Object] The processed text with replacements or original object
-    def apply_replacements(text)
-      result = text.to_s
-
-      @replacements.each do |key, value|
-        result = result.gsub(/\{#{key}\}/, value.to_s)
-      end
-      result
-    end
-  end
-end