scraper_utils 0.8.2 → 0.8.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5103f6e00bdb69c631084a1c6bc939bb7380b0dcd342e21c61eb4b4756482aa9
-  data.tar.gz: '068e221220371c3e27379ccccd7f784db108985e02273da92eca9d1d94b90620'
+  metadata.gz: 858cdc2f2f55b23ea10a09629ff347b2a6833c8efd1f89851168db7ffb46d9f8
+  data.tar.gz: f6d17f7a327dd3698b12e4da7772037e58cb76201d3271def86f195a68e97b03
 SHA512:
-  metadata.gz: f6b04d20acfe5944f24b1c3a86816d817088d85c2bb208699eee0d88da1f9927d645d84ed3170483843cb8d34f09d06db18982f660b31392853b696a4c3239ed
-  data.tar.gz: 0ab144cb6f9b9da5df3476f441b3701e254ddf18b264a181394da7e535a53ec234cfb6f75d2b50a0b0557fdb435b663427356178adee8505c5e2633f341d8676
+  metadata.gz: 0605eaa66aa317a0076d7c87ebdb883e274bd0fd44fd53ab28219a09a963c78decd0a086368e5b0e01748c0c67cd9ef4a84b48274794ad6bb5ea6e9dcdff4745
+  data.tar.gz: 511175ebaed815b6b851f804029b33c8ebbe553623d63556d78bb49f5be1fe4d52e5d9a78f27a75f3423ea90df7aad244acb43415b3990f86f78a9986438e21c

data/.gitignore

@@ -1,9 +1,44 @@
+# Scraper specific example
+
+# Ignore output of scraper
+data.sqlite
+
+# Don't save limited scraper runs
+/spec/cassettes/limited_to_*
+/spec/expected/limited_to_*
+
+# ---------------------------
+# Gem ignores 
 *.gem
+/pkg/
+/InstalledFiles
+
+# REMOVE PARAGRAPH IF USING WITH SCRAPERS:
+# for a library or gem, you might want to ignore these files since the code is
+# intended to run in multiple environments; otherwise, check them in:
+Gemfile.lock
+.ruby-version
+.ruby-gemset
+
+# ---------------------------
+# General ignore 
+
+# rspec failure tracking
+.rspec_status
+.release
+.bundle
+.profile.d
+bin
+vendor
+tmp
+.basher/bash
+Procfile
+
+# Spec coverage
+/coverage/
+
 *.rbc
 /.config
-/coverage/
-/InstalledFiles
-/pkg/
 /spec/reports/
 /spec/examples.txt
 /test/tmp/
@@ -64,12 +99,6 @@ build-iPhoneSimulator/
 /vendor/bundle
 /lib/bundler/man/
 
-# for a library or gem, you might want to ignore these files since the code is
-# intended to run in multiple environments; otherwise, check them in:
-Gemfile.lock
-.ruby-version
-.ruby-gemset
-
 # unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
 .rvmrc
 
@@ -78,3 +107,4 @@ Gemfile.lock
 
 # rspec reports
 /.rspec_status
+

data/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 0.9.0 - 2025-07-01
+
+**Significant cleanup - removed code we ended up not using as none of the councils are actually concerned about server load**
+
+* Refactored example code into simple callable methods
+* Expand test for geocodeable addresses to include comma between postcode and state at the end of the address.
+
+### Added
+- `ScraperUtils::SpecSupport.validate_addresses_are_geocodable!` - validates percentage of geocodable addresses
+- `ScraperUtils::SpecSupport.validate_descriptions_are_reasonable!` - validates percentage of reasonable descriptions
+- `ScraperUtils::SpecSupport.validate_uses_one_valid_info_url!` - validates single global info_url usage and availability
+- `ScraperUtils::SpecSupport.validate_info_urls_have_expected_details!` - validates info_urls contain expected content
+- `ScraperUtils::MathsUtils.fibonacci_series` - generates fibonacci sequence up to max value
+- `bot_check_expected` parameter to info_url validation methods for handling reCAPTCHA/Cloudflare protection
+
+### Fixed
+- Typo in `geocodable?` method debug output (`has_suburb_stats` → `has_suburb_states`) 
+- Code example in `docs/enhancing_specs.md`
+
+### Updated
+- Code example in `docs/enhancing_specs.md` to reflect new support methods
+- geocodeable? test is simpler - it requires 
+  - a street type, unit or lot, 
+  - an uppercase word (assumed to be a suburb) or postcode and 
+  - a state
+
+### Removed
+- Unsued CycleUtils
+- Unused DateRangeUtils
+- Unused RandomizeUtils
+- Unused Scheduling (Fiber and Threads)
+- Unused Compliant mode, delays for Agent (Agent is configured with an agent string)
+- Unused MechanizeActions 
+
 ## 0.8.2 - 2025-05-07
 
 * Ignore blank dates supplied when validating rather than complain they are not valid

data/README.md

@@ -3,43 +3,6 @@ ScraperUtils (Ruby)
 
 Utilities to help make planningalerts scrapers, especially multis, easier to develop, run and debug.
 
-For Server Administrators
--------------------------
-
-The ScraperUtils library is designed to be a respectful citizen of the web. If you're a server administrator and notice
-our scraper accessing your systems, here's what you should know:
-
-### We play nice with your servers
-
-Our goal is to access public planning information with minimal impact on your services. The following features are on by
-default:
-
-- **Limit server load**:
-    - We limit the max load we present to your server to less than a half of one of your cpu cores
-        - The more loaded your server is, the longer we wait between requests!
-    - We respect Crawl-delay from robots.txt (see section below), so you can tell us an acceptable rate
-    - Scarper developers can
-        - reduce the max_load we present to your server even further
-        - add random extra delays to give your server a chance to catch up with background tasks
-
-- **Identify themselves**: Our user agent clearly indicates who we are and provides a link to the project repository:
-  `Mozilla/5.0 (compatible; ScraperUtils/0.2.0 2025-02-22; +https://github.com/ianheggie-oaf/scraper_utils)`
-
-### How to Control Our Behavior
-
-Our scraper utilities respect the standard server **robots.txt** control mechanisms (by default).
-To control our access:
-
-- Add a section for our user agent: `User-agent: ScraperUtils`
-- Set a crawl delay, eg: `Crawl-delay: 20`
-- If needed specify disallowed paths: `Disallow: /private/`
-
-For Scraper Developers
-----------------------
-
-We provide utilities to make developing, running and debugging your scraper easier in addition to the base utilities
-mentioned above.
-
 ## Installation & Configuration
 
 Add to [your scraper's](https://www.planningalerts.org.au/how_to_write_a_scraper) Gemfile:
@@ -57,27 +20,9 @@ see {file:docs/getting_started.md Getting Started guide}
 ### Well-Behaved Web Client
 
 - Configure Mechanize agents with sensible defaults
-- Automatic rate limiting based on server response times
-- Supports robots.txt and crawl-delay directives
 - Supports extra actions required to get to results page
 - {file:docs/mechanize_utilities.md Learn more about Mechanize utilities}
 
-### Optimize Server Load
-
-- Intelligent date range selection (reduce server load by up to 60%)
-- Cycle utilities for rotating search parameters
-- {file:docs/reducing_server_load.md Learn more about reducing server load}
-
-### Improve Scraper Efficiency
-
-- Interleaves requests to optimize run time
-    - {file:docs/interleaving_requests.md Learn more about interleaving requests}
-- Use {ScraperUtils::Scheduler.execute_request} so Mechanize network requests will be performed by threads in parallel
-    - {file:docs/parallel_requests.md Parallel Request} - see Usage section for installation instructions
-- Randomize processing order for more natural request patterns
-    - {file:docs/randomizing_requests.md Learn more about randomizing requests} - see Usage section for installation
-      instructions
-
 ### Error Handling & Quality Monitoring
 
 - Record-level error handling with appropriate thresholds
@@ -90,6 +35,13 @@ see {file:docs/getting_started.md Getting Started guide}
 - Simple logging with authority context
 - {file:docs/debugging.md Learn more about debugging}
 
+### Spec Validation
+
+- Validate geocodable addresses and reasonable descriptions
+- Check info URL availability and content
+- Support for bot protection detection
+- {file:docs/enhancing_specs.md Learn more about spec validation}
+
 ## API Documentation
 
 Complete API documentation is available at [scraper_utils | RubyDoc.info](https://rubydoc.info/gems/scraper_utils).

data/docs/enhancing_specs.md

@@ -1,9 +1,13 @@
 # Enhancing specs
 
-ScraperUtils provides two methods to help with checking results
+ScraperUtils provides validation methods to help with checking scraping results:
 
-* `ScraperUtils::SpecSupport.geocodable?`
-* `ScraperUtils::SpecSupport.reasonable_description?`
+* `ScraperUtils::SpecSupport.geocodable?` - Check if an address is likely to be geocodable
+* `ScraperUtils::SpecSupport.reasonable_description?` - Check if a description looks reasonable
+* `ScraperUtils::SpecSupport.validate_addresses_are_geocodable!` - Validate percentage of geocodable addresses
+* `ScraperUtils::SpecSupport.validate_descriptions_are_reasonable!` - Validate percentage of reasonable descriptions
+* `ScraperUtils::SpecSupport.validate_uses_one_valid_info_url!` - Validate single global info_url usage and availability
+* `ScraperUtils::SpecSupport.validate_info_urls_have_expected_details!` - Validate info_urls contain expected content
 
 ## Example Code:
 
@@ -14,9 +18,16 @@ require "timecop"
 require_relative "../scraper"
 
 RSpec.describe Scraper do
+  # Authorities that use bot protection (reCAPTCHA, Cloudflare, etc.) on detail pages (info_url)
+  AUTHORITIES_WITH_BOT_PROTECTION = %i[
+    some_authority
+    another_authority
+  ].freeze
+
   describe ".scrape" do
     def test_scrape(authority)
-      File.delete("./data.sqlite") if File.exist?("./data.sqlite")
+      ScraperWiki.close_sqlite
+      FileUtils.rm_f("data.sqlite")
 
       VCR.use_cassette(authority) do
         date = Date.new(2025, 4, 15)
@@ -44,48 +55,20 @@ RSpec.describe Scraper do
 
       expect(results).to eq expected
 
-      geocodable = results
-                     .map { |record| record["address"] }
-                     .uniq
-                     .count { |text| SpecHelper.geocodable? text }
-      puts "Found #{geocodable} out of #{results.count} unique geocodable addresses " \
-        "(#{(100.0 * geocodable / results.count).round(1)}%)"
-      expect(geocodable).to be > (0.7 * results.count)
-
-      descriptions = results
-                       .map { |record| record["description"] }
-                       .uniq
-                       .count do |text|
-        selected = SpecHelper.reasonable_description? text
-        puts "  description: #{text} is not reasonable" if ENV["DEBUG"] && !selected
-        selected
-      end
-      puts "Found #{descriptions} out of #{results.count} unique reasonable descriptions " \
-             "(#{(100.0 * descriptions / results.count).round(1)}%)"
-      expect(descriptions).to be > (0.55 * results.count)
-
-      info_urls = results
-                  .map { |record| record["info_url"] }
-                  .uniq
-                  .count { |text| text.to_s.match(%r{\Ahttps?://}) }
-      puts "Found #{info_urls} out of #{results.count} unique info_urls " \
-             "(#{(100.0 * info_urls / results.count).round(1)}%)"
-      expect(info_urls).to be > (0.7 * results.count) if info_urls != 1
-
-      VCR.use_cassette("#{authority}.info_urls") do
-        results.each do |record|
-          info_url = record["info_url"]
-          puts "Checking info_url #{info_url} #{info_urls > 1 ? ' has expected details' : ''} ..."
-          response = Net::HTTP.get_response(URI(info_url))
-
-          expect(response.code).to eq("200")
-          # If info_url is the same for all records, then it won't have details
-          break if info_urls == 1
-
-          expect(response.body).to include(record["council_reference"])
-          expect(response.body).to include(record["address"])
-          expect(response.body).to include(record["description"])
-        end
+      # Validate addresses are geocodable (70% minimum with 3 records variation)
+      ScraperUtils::SpecSupport.validate_addresses_are_geocodable!(results, percentage: 70, variation: 3)
+
+      # Validate descriptions are reasonable (55% minimum with 3 records variation)
+      ScraperUtils::SpecSupport.validate_descriptions_are_reasonable!(results, percentage: 55, variation: 3)
+
+      # Validate info_urls based on authority configuration
+      global_info_url = Scraper::AUTHORITIES[authority][:info_url]
+      bot_check_expected = AUTHORITIES_WITH_BOT_PROTECTION.include?(authority)
+
+      if global_info_url
+        ScraperUtils::SpecSupport.validate_uses_one_valid_info_url!(results, global_info_url, bot_check_expected: bot_check_expected)
+      else
+        ScraperUtils::SpecSupport.validate_info_urls_have_expected_details!(results, percentage: 75, variation: 3, bot_check_expected: bot_check_expected)
       end
     end
 
@@ -96,5 +79,61 @@ RSpec.describe Scraper do
     end
   end
 end
+```
+
+## Individual validation methods:
+
+### Address validation
+
+```ruby
+# Check if single address is geocodable
+ScraperUtils::SpecSupport.geocodable?('123 Smith Street, Sydney NSW 2000')
+
+# Validate percentage of geocodable addresses
+ScraperUtils::SpecSupport.validate_addresses_are_geocodable!(results, percentage: 50, variation: 3)
+```
+
+### Description validation
+
+```ruby
+# Check if single description is reasonable
+ScraperUtils::SpecSupport.reasonable_description?('Construction of new building')
+
+# Validate percentage of reasonable descriptions
+ScraperUtils::SpecSupport.validate_descriptions_are_reasonable!(results, percentage: 50, variation: 3)
+```
+
+### Info URL validation
+
+```ruby
+# For authorities with global info_url
+ScraperUtils::SpecSupport.validate_uses_one_valid_info_url!(results, 'https://example.com/search')
+
+# For authorities with bot protection
+ScraperUtils::SpecSupport.validate_uses_one_valid_info_url!(results, 'https://example.com/search', bot_check_expected: true)
+
+# For authorities with unique info_urls per record
+ScraperUtils::SpecSupport.validate_info_urls_have_expected_details!(results, percentage: 75, variation: 3)
+
+# For authorities with unique info_urls that may have bot protection
+ScraperUtils::SpecSupport.validate_info_urls_have_expected_details!(results, percentage: 75, variation: 3, bot_check_expected: true)
+```
+
+## Bot Protection Handling
+
+The `bot_check_expected` parameter allows validation methods to accept bot protection as valid responses:
+
+**When bot protection is detected:**
+
+- HTTP status codes: 403 (Forbidden), 429 (Too Many Requests)
+- Content indicators: "recaptcha", "cloudflare", "are you human", "bot detection", "security check", "verify you are
+  human", "access denied", "blocked", "captcha"
+
+**Usage patterns:**
+
+- Set `bot_check_expected: false` (default) - requires 200 responses, fails on bot protection
+- Set `bot_check_expected: true` - accepts bot protection as valid, useful for authorities known to use anti-bot
+  measures
 
-```
+All validation methods accept `percentage` (minimum percentage required) and `variation` (additional tolerance)
+parameters for consistent configuration.

data/docs/example_custom_Rakefile

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+# Remember to add scraper_utils to Gemfile!
+
+require "fileutils"
+
+desc "Run scraper and validate results"
+task :test do
+  Rake::Task[:scrape].invoke
+  Rake::Task[:validate].invoke
+end
+
+desc "Run the scraper"
+task :scrape do
+  puts "Running scraper..."
+  FileUtils.rm_f("data.sqlite")
+
+  system("bundle exec ruby scraper.rb") || abort("Scraper failed")
+end
+
+desc "Validate scraped data"
+task :validate do
+  puts "Validating scraped data..."
+
+  unless File.exist?("data.sqlite")
+    abort("No data.sqlite found - run scraper first")
+  end
+
+  system("validate_scraper_data") || abort("Validation failed")
+end
+
+desc "Clean up generated files"
+task :clean do
+  FileUtils.rm_f("data.sqlite")
+  puts "Cleaned up data.sqlite"
+end
+
+task default: :test

data/docs/example_dot_scraper_validation.yml

@@ -0,0 +1,23 @@
+# Scraper validation configuration
+# Copy this to .scraper_validation.yml and customize as needed
+
+# Database settings
+database: "data.sqlite"
+
+# Address validation (percentage of addresses that should be geocodable)
+geocodable_percentage: 70
+geocodable_variation: 3
+
+# Description validation (percentage of descriptions that should be reasonable)
+description_percentage: 55
+description_variation: 3
+
+# Info URL validation (percentage of detail checks that should pass)
+info_url_percentage: 75
+info_url_variation: 3
+
+# Bot protection (set to true if info URLs use reCAPTCHA/Cloudflare)
+bot_check_expected: false
+
+# Global info URL (leave blank for auto-detection)
+# global_info_url: "https://example.com/search"

data/docs/mechanize_utilities.md

@@ -36,9 +36,6 @@ Note - compliant mode forces max_load to be set to a value no greater than 50.
 ```ruby
 ScraperUtils::MechanizeUtils::AgentConfig.configure do |config|
   config.default_timeout = ENV.fetch('MORPH_TIMEOUT', DEFAULT_TIMEOUT).to_i # 60
-  config.default_compliant_mode = ENV.fetch('MORPH_NOT_COMPLIANT', nil).to_s.empty? # true
-  config.default_random_delay = ENV.fetch('MORPH_RANDOM_DELAY', DEFAULT_RANDOM_DELAY).to_i # 0
-  config.default_max_load = ENV.fetch('MORPH_MAX_LOAD',DEFAULT_MAX_LOAD).to_f # 50.0
   config.default_disable_ssl_certificate_check = !ENV.fetch('MORPH_DISABLE_SSL_CHECK', nil).to_s.empty? # false
   config.default_australian_proxy = !ENV.fetch('MORPH_USE_PROXY', nil).to_s.empty? # false
   config.default_user_agent = ENV.fetch('MORPH_USER_AGENT', nil) # Uses Mechanize user agent

data/docs/testing_custom_scrapers.md

@@ -0,0 +1,74 @@
+# Testing Custom Scrapers
+
+Custom scrapers often lack proper specs but still need quality validation. This gem provides a simple executable to
+validate your scraped data.
+
+## Setup
+
+Add to your `Gemfile`:
+
+```ruby
+gem "scraper_utils", "~> 0.8.2"
+gem "rake", "~> 13.0"
+```
+
+Copy the example files to your scraper directory:
+
+- `docs/example_custom_Rakefile` → `Rakefile`
+- `docs/example_dot_scraper_validation.yml` → `.scraper_validation.yml`
+
+## Usage
+
+### Command Line
+
+Once you have run the scraper (`bundle exec ruby scraper.rb`) you can manually run the command:
+
+```bash
+# Run with defaults
+validate_scraper_data
+
+# Custom thresholds
+validate_scraper_data --geocodable-percentage 80 --description-percentage 60
+
+# For scrapers with bot protection
+validate_scraper_data --bot-check-expected
+```
+
+### Rake Tasks
+
+```bash
+bundle exec rake           # Run scraper then validate (default)
+bundle exec rake test      # Same as above
+bundle exec rake scrape    # Just run scraper
+bundle exec rake validate  # Just validate existing data
+bundle exec rake clean     # Remove data.sqlite
+```
+
+## Configuration
+
+Edit `.scraper_validation.yml` to set validation thresholds:
+
+```yaml
+geocodable_percentage: 70    # Min % of geocodable addresses
+description_percentage: 55   # Min % of reasonable descriptions
+info_url_percentage: 75      # Min % of info URL detail checks
+bot_check_expected: false    # Set true when detail pages have reCAPTCHA/Cloudflare bot checks
+```
+
+## What Gets Validated
+
+- **Addresses**: Checks for proper Australian address format (street type, state, postcode)
+- **Descriptions**: Ensures descriptions aren't placeholders and have 3+ words
+- **Info URLs**: Validates URLs return status 200 and the page contains expected details
+- **Global URLs**: Auto-detects if all records use the same info URL and validates it's accessible
+
+## Integration with CI
+
+Add to your CI pipeline:
+
+```bash
+bundle install
+rake test
+```
+
+The validation will fail with clear error messages if data quality is insufficient, helping catch scraping issues early.

data/exe/validate_scraper_data

@@ -0,0 +1,150 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "optparse"
+require "sqlite3"
+require "yaml"
+require "scraper_utils"
+
+# Default validation options
+options = {
+  database: "data.sqlite",
+  geocodable_percentage: 50,
+  geocodable_variation: 3,
+  description_percentage: 50,
+  description_variation: 3,
+  info_url_percentage: 75,
+  info_url_variation: 3,
+  bot_check_expected: false,
+  global_info_url: nil
+}
+
+# Load config file if it exists
+config_file = ".scraper_validation.yml"
+if File.exist?(config_file)
+  begin
+    config = YAML.safe_load(File.read(config_file), symbolize_names: true)
+    options.merge!(config) if config
+    puts "Loaded config from #{config_file}"
+  rescue => e
+    puts "Warning: Could not load #{config_file}: #{e.message}"
+  end
+end
+
+OptionParser.new do |opts|
+  opts.banner = "Usage: validate_scraper_data [options]"
+
+  opts.on("-d", "--database PATH", "SQLite database path (default: data.sqlite)") do |db|
+    options[:database] = db
+  end
+
+  opts.on("-g", "--geocodable-percentage N", Integer, "Min percentage of geocodable addresses (default: 50)") do |n|
+    options[:geocodable_percentage] = n
+  end
+
+  opts.on("-r", "--description-percentage N", Integer, "Min percentage of reasonable descriptions (default: 50)") do |n|
+    options[:description_percentage] = n
+  end
+
+  opts.on("-u", "--info-url-percentage N", Integer, "Min percentage for info URL validation (default: 75)") do |n|
+    options[:info_url_percentage] = n
+  end
+
+  opts.on("-v", "--variation N", Integer, "Variation tolerance for all validations (default: 3)") do |n|
+    options[:geocodable_variation] = n
+    options[:description_variation] = n
+    options[:info_url_variation] = n
+  end
+
+  opts.on("-b", "--bot-check-expected", "Expect bot protection on info URLs") do
+    options[:bot_check_expected] = true
+  end
+
+  opts.on("-i", "--global-info-url URL", "Validate all records use this global info URL (auto-detected if all URLs are the same)") do |url|
+    options[:global_info_url] = url
+  end
+
+  opts.on("-c", "--config FILE", "Load config from YAML file (default: .scraper_validation.yml)") do |file|
+    config_file = file
+  end
+
+  opts.on("-h", "--help", "Show this help") do
+    puts opts
+    exit
+  end
+end.parse!
+
+# Check database exists
+unless File.exist?(options[:database])
+  puts "Error: Database file '#{options[:database]}' not found"
+  exit 1
+end
+
+# Read data from SQLite
+begin
+  db = SQLite3::Database.new(options[:database])
+  db.results_as_hash = true
+  results = db.execute("SELECT * FROM data ORDER BY council_reference")
+  db.close
+rescue SQLite3::Exception => e
+  puts "Error reading database: #{e.message}"
+  exit 1
+end
+
+if results.empty?
+  puts "No data found in database"
+  exit 1
+end
+
+puts "Validating #{results.count} records from #{options[:database]}..."
+
+# Auto-detect global info URL if not specified
+if options[:global_info_url].nil?
+  info_urls = results.map { |record| record["info_url"] }.compact.uniq
+  if info_urls.size == 1
+    options[:global_info_url] = info_urls.first
+    puts "Auto-detected global info_url: #{options[:global_info_url]}"
+  end
+end
+
+puts
+
+begin
+  # Validate addresses are geocodable
+  ScraperUtils::SpecSupport.validate_addresses_are_geocodable!(
+    results,
+    percentage: options[:geocodable_percentage],
+    variation: options[:geocodable_variation]
+  )
+
+  # Validate descriptions are reasonable
+  ScraperUtils::SpecSupport.validate_descriptions_are_reasonable!(
+    results,
+    percentage: options[:description_percentage],
+    variation: options[:description_variation]
+  )
+
+  # Validate info URLs
+  if options[:global_info_url]
+    ScraperUtils::SpecSupport.validate_uses_one_valid_info_url!(
+      results,
+      options[:global_info_url],
+      bot_check_expected: options[:bot_check_expected]
+    )
+  else
+    ScraperUtils::SpecSupport.validate_info_urls_have_expected_details!(
+      results,
+      percentage: options[:info_url_percentage],
+      variation: options[:info_url_variation],
+      bot_check_expected: options[:bot_check_expected]
+    )
+  end
+
+  puts
+  puts "✅ All validations passed!"
+
+rescue RuntimeError => e
+  puts
+  puts "❌ Validation failed: #{e.message}"
+  exit 1
+end