scraper_utils 0.5.1 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 28f415290516d19f6ffc95a7d349a6ed269de987a0ffe45ed512ff29bfc82902
-  data.tar.gz: 9d337d1145754bf2375f4d2e18d89da7c21231d7b9b279bc51340f92781caa35
+  metadata.gz: da26385a1d788bc9ad9d725f0eaefe233d4b70a8bf9aeab0af3168041adc0bc2
+  data.tar.gz: '02892db893cc706ec67845bde0a05d80b89cdfc2d48759b2e66574e6f87b0031'
 SHA512:
-  metadata.gz: 8a3050451f512b2f77cf9cd806fc1602d6502b247f24248d93cfc12dea47bf5d7f02bd9be5453c0da7ee7b8acc0e3ee32cd375b52e04f0f366c2174ea7320bd9
-  data.tar.gz: 17befcb8b9305536385ddf6772aeee038df34bd42ce197c4951836bb396e1db9d3dabece2a543ece6f60e331ca244ef1f660f1b111af17d79703a2a783801183
+  metadata.gz: b8101b0b0d2ed1d775de54f0e8bac5a1a22ca6f540cea2752de76218a4915c325ec7569ac76a719bf540474f54123cb32f700635160cdbabdcc68679ac33c2e4
+  data.tar.gz: ae1e5d72f45b077f0525e62dc0399f9bd6f519d52222518e95f57f8f45e03575f3b8b909fb02b030a27b33bfccaefec74bb143d9d13ff1bcfe094ceaa8e369d3

data/.yardopts

@@ -0,0 +1,5 @@
+--files docs/*.rb
+--files docs/*.md
+--files CHANGELOG.md
+--files LICENSE.txt
+--readme README.md

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## 0.7.0 - 2025-04-15
+
+* Added Spec helpers and associated doc: `docs/enhancing_specs.md`
+  * `ScraperUtils::SpecSupport.geocodable?`
+  * `ScraperUtils::SpecSupport.reasonable_description?`
+
+## 0.6.1 - 2025-03-28
+
+* Changed DEFAULT_MAX_LOAD to 50.0 as we are overestimating the load we present as network latency is included
+* Correct documentation of spec_helper extra lines
+* Fix misc bugs found in use
+
+## 0.6.0 - 2025-03-16
+
+* Add threads for more efficient scraping
+* Adjust defaults for more efficient scraping, retaining just response based delays by default
+* Correct and simplify date range utilities so everything is checked at least `max_period` days
+* Release Candidate for v1.0.0, subject to testing in production
+
 ## 0.5.1 - 2025-03-05
 
 * Remove duplicated example code in docs

data/GUIDELINES.md

@@ -47,7 +47,8 @@ but if the file is bad, just treat it as missing.
 
 ## Testing Strategies
 
-* Avoid mocking unless really needed, instead
+* AVOID mocking unless really needed (and REALLY avoid mocking your own code), instead
+  * Consider if you can change your own code, whilst keeping it simple, to make it easier to test
   * instantiate a real object to use in the test
   * use mocking facilities provided by the gem (eg Mechanize, Aws etc)
   * use integration tests with WebMock for simple external sites or VCR for more complex.

data/Gemfile

@@ -32,5 +32,6 @@ gem "simplecov", platform && (platform == :heroku16 ? "~> 0.18.0" : "~> 0.22.0")
 gem "simplecov-console"
 gem "terminal-table"
 gem "webmock", platform && (platform == :heroku16 ? "~> 3.14.0" : "~> 3.19.0")
+gem "yard"
 
 gemspec

data/IMPLEMENTATION.md

@@ -31,3 +31,42 @@ puts "Pre Connect request: #{request.inspect}" if ENV["DEBUG"]
 - Externalize configuration to improve testability
 - Keep shared logic in the main class
 - Decisions / information specific to just one class, can be documented there, otherwise it belongs here
+
+## Testing Directory Structure
+
+Our test directory structure reflects various testing strategies and aspects of the codebase:
+
+### API Context Directories
+- `spec/scraper_utils/fiber_api/` - Tests functionality called from within worker fibers
+- `spec/scraper_utils/main_fiber/` - Tests functionality called from the main fiber's perspective
+- `spec/scraper_utils/thread_api/` - Tests functionality called from within worker threads
+
+### Utility Classes
+- `spec/scraper_utils/mechanize_utils/` - Tests for `lib/scraper_utils/mechanize_utils/*.rb` files
+- `spec/scraper_utils/scheduler/` - Tests for `lib/scraper_utils/scheduler/*.rb` files
+- `spec/scraper_utils/scheduler2/` - FIXME: remove duplicate tests and merge to `spec/scraper_utils/scheduler/` unless > 200 lines
+
+### Integration vs Unit Tests
+- `spec/scraper_utils/integration/` - Tests that focus on the integration between components
+  - Name tests after the most "parent-like" class of the components involved
+
+### Special Configuration Directories
+These specs check the options we use when things go wrong in production
+
+- `spec/scraper_utils/no_threads/` - Tests with threads disabled (`MORPH_DISABLE_THREADS=1`)
+- `spec/scraper_utils/sequential/` - Tests with exactly one worker (`MORPH_MAX_WORKERS=1`)
+
+### Directories to break up large specs
+Keep specs less than 200 lines long
+
+- `spec/scraper_utils/replacements` - Tests for replacements in MechanizeActions
+- `spec/scraper_utils/replacements2` - FIXME: remove duplicate tests and merge to `spec/scraper_utils/replacements/`?
+- `spec/scraper_utils/selectors` - Tests the various node selectors available in MechanizeActions
+- `spec/scraper_utils/selectors2` - FIXME: remove duplicate tests and merge to `spec/scraper_utils/selectors/`?
+
+### General Testing Guidelines
+- Respect fiber and thread context validation - never mock the objects under test
+- Structure tests to run in the appropriate fiber context
+- Use real fibers, threads and operations rather than excessive mocking
+- Ensure proper cleanup of resources in both success and error paths
+- ASK when unsure which (yard doc, spec or code) is wrong as I don't always follow the "write specs first" strategy

data/README.md

@@ -9,28 +9,30 @@ For Server Administrators
 The ScraperUtils library is designed to be a respectful citizen of the web. If you're a server administrator and notice
 our scraper accessing your systems, here's what you should know:
 
-### How to Control Our Behavior
-
-Our scraper utilities respect the standard server **robots.txt** control mechanisms (by default).
-To control our access:
-
-- Add a section for our user agent: `User-agent: ScraperUtils` (default)
-- Set a crawl delay, eg: `Crawl-delay: 20`
-- If needed specify disallowed paths: `Disallow: /private/`
-
 ### We play nice with your servers
 
 Our goal is to access public planning information with minimal impact on your services. The following features are on by
 default:
 
+- **Limit server load**:
+    - We limit the max load we present to your server to less than a half of one of your cpu cores
+        - The more loaded your server is, the longer we wait between requests!
+    - We respect Crawl-delay from robots.txt (see section below), so you can tell us an acceptable rate
+    - Scarper developers can
+        - reduce the max_load we present to your server even further
+        - add random extra delays to give your server a chance to catch up with background tasks
+
 - **Identify themselves**: Our user agent clearly indicates who we are and provides a link to the project repository:
   `Mozilla/5.0 (compatible; ScraperUtils/0.2.0 2025-02-22; +https://github.com/ianheggie-oaf/scraper_utils)`
 
-- **Limit server load**:
-    - We wait double your response time before making another request to avoid being a significant load on your server
-    - We also randomly add extra delays to give your server a chance to catch up with background tasks
+### How to Control Our Behavior
+
+Our scraper utilities respect the standard server **robots.txt** control mechanisms (by default).
+To control our access:
 
-We also provide scraper developers other features to reduce overall load as well.
+- Add a section for our user agent: `User-agent: ScraperUtils`
+- Set a crawl delay, eg: `Crawl-delay: 20`
+- If needed specify disallowed paths: `Disallow: /private/`
 
 For Scraper Developers
 ----------------------
@@ -40,14 +42,15 @@ mentioned above.
 
 ## Installation & Configuration
 
-Add to your [scraper's](https://www.planningalerts.org.au/how_to_write_a_scraper) Gemfile:
+Add to [your scraper's](https://www.planningalerts.org.au/how_to_write_a_scraper) Gemfile:
 
 ```ruby
 gem "scraperwiki", git: "https://github.com/openaustralia/scraperwiki-ruby.git", branch: "morph_defaults"
 gem 'scraper_utils'
 ```
 
-For detailed setup and configuration options, see the [Getting Started guide](docs/getting_started.md).
+For detailed setup and configuration options,
+see {file:docs/getting_started.md Getting Started guide}
 
 ## Key Features
 
@@ -57,20 +60,23 @@ For detailed setup and configuration options, see the [Getting Started guide](do
 - Automatic rate limiting based on server response times
 - Supports robots.txt and crawl-delay directives
 - Supports extra actions required to get to results page
-- [Learn more about Mechanize utilities](docs/mechanize_utilities.md)
+- {file:docs/mechanize_utilities.md Learn more about Mechanize utilities}
 
 ### Optimize Server Load
 
 - Intelligent date range selection (reduce server load by up to 60%)
 - Cycle utilities for rotating search parameters
-- [Learn more about reducing server load](docs/reducing_server_load.md)
+- {file:docs/reducing_server_load.md Learn more about reducing server load}
 
 ### Improve Scraper Efficiency
 
-- Interleave requests to optimize run time
-- [Learn more about interleaving requests](docs/interleaving_requests.md)
+- Interleaves requests to optimize run time
+    - {file:docs/interleaving_requests.md Learn more about interleaving requests}
+- Use {ScraperUtils::Scheduler.execute_request} so Mechanize network requests will be performed by threads in parallel
+    - {file:docs/parallel_requests.md Parallel Request} - see Usage section for installation instructions
 - Randomize processing order for more natural request patterns
-- [Learn more about randomizing requests](docs/randomizing_requests.md)
+    - {file:docs/randomizing_requests.md Learn more about randomizing requests} - see Usage section for installation
+      instructions
 
 ### Error Handling & Quality Monitoring
 
@@ -82,11 +88,11 @@ For detailed setup and configuration options, see the [Getting Started guide](do
 
 - Enhanced debugging utilities
 - Simple logging with authority context
-- [Learn more about debugging](docs/debugging.md)
+- {file:docs/debugging.md Learn more about debugging}
 
 ## API Documentation
 
-Complete API documentation is available at [RubyDoc.info](https://rubydoc.info/gems/scraper_utils).
+Complete API documentation is available at [scraper_utils | RubyDoc.info](https://rubydoc.info/gems/scraper_utils).
 
 ## Ruby Versions
 
@@ -105,7 +111,7 @@ To install this gem onto your local machine, run `bundle exec rake install`.
 ## Contributing
 
 Bug reports and pull requests with working tests are welcome
-on [GitHub](https://github.com/ianheggie-oaf/scraper_utils).
+on [ianheggie-oaf/scraper_utils | GitHub](https://github.com/ianheggie-oaf/scraper_utils).
 
 ## License
 

data/SPECS.md

@@ -6,7 +6,13 @@ installation and usage notes in `README.md`.
 
 ASK for clarification of any apparent conflicts with IMPLEMENTATION, GUIDELINES or project instructions.
 
-## Core Design Principles
+Core Design Principles
+----------------------
+
+## Coding Style and Complexity
+- KISS (Keep it Simple and Stupid) is a guiding principle:
+  - Simple: Design and implement with as little complexity as possible while still achieving the desired functionality
+  - Stupid: Should be easy to diagnose and repair with basic tooling
 
 ### Error Handling
 - Record-level errors abort only that record's processing
@@ -23,3 +29,9 @@ ASK for clarification of any apparent conflicts with IMPLEMENTATION, GUIDELINES
 - Ensure components are independently testable
 - Avoid timing-based tests in favor of logic validation
 - Keep test scenarios focused and under 20 lines
+
+#### Fiber and Thread Testing
+- Test in appropriate fiber/thread context using API-specific directories
+- Validate cooperative concurrency with real fibers rather than mocks
+- Ensure tests for each context: main fiber, worker fibers, and various thread configurations
+- Test special configurations (no threads, no fibers, sequential) in dedicated directories

data/bin/rspec

@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+#
+# This file was generated by Bundler.
+#
+# The application 'rspec' is installed as part of a gem, and
+# this file is here to facilitate running it.
+#
+
+ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)
+
+bundle_binstub = File.expand_path("bundle", __dir__)
+
+if File.file?(bundle_binstub)
+  if File.read(bundle_binstub, 300).include?("This file was generated by Bundler")
+    load(bundle_binstub)
+  else
+    abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run.
+Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.")
+  end
+end
+
+require "rubygems"
+require "bundler/setup"
+
+load Gem.bin_path("rspec-core", "rspec")

data/docs/enhancing_specs.md

@@ -0,0 +1,100 @@
+# Enhancing specs
+
+ScraperUtils provides two methods to help with checking results
+
+* `ScraperUtils::SpecSupport.geocodable?`
+* `ScraperUtils::SpecSupport.reasonable_description?`
+
+## Example Code:
+
+```ruby
+# frozen_string_literal: true
+
+require "timecop"
+require_relative "../scraper"
+
+RSpec.describe Scraper do
+  describe ".scrape" do
+    def test_scrape(authority)
+      File.delete("./data.sqlite") if File.exist?("./data.sqlite")
+
+      VCR.use_cassette(authority) do
+        date = Date.new(2025, 4, 15)
+        Timecop.freeze(date) do
+          Scraper.scrape([authority], 1)
+        end
+      end
+
+      expected = if File.exist?("spec/expected/#{authority}.yml")
+                   YAML.safe_load(File.read("spec/expected/#{authority}.yml"))
+                 else
+                   []
+                 end
+      results = ScraperWiki.select("* from data order by council_reference")
+
+      ScraperWiki.close_sqlite
+
+      if results != expected
+        # Overwrite expected so that we can compare with version control
+        # (and maybe commit if it is correct)
+        File.open("spec/expected/#{authority}.yml", "w") do |f|
+          f.write(results.to_yaml)
+        end
+      end
+
+      expect(results).to eq expected
+
+      geocodable = results
+                     .map { |record| record["address"] }
+                     .uniq
+                     .count { |text| SpecHelper.geocodable? text }
+      puts "Found #{geocodable} out of #{results.count} unique geocodable addresses " \
+        "(#{(100.0 * geocodable / results.count).round(1)}%)"
+      expect(geocodable).to be > (0.7 * results.count)
+
+      descriptions = results
+                       .map { |record| record["description"] }
+                       .uniq
+                       .count do |text|
+        selected = SpecHelper.reasonable_description? text
+        puts "  description: #{text} is not reasonable" if ENV["DEBUG"] && !selected
+        selected
+      end
+      puts "Found #{descriptions} out of #{results.count} unique reasonable descriptions " \
+             "(#{(100.0 * descriptions / results.count).round(1)}%)"
+      expect(descriptions).to be > (0.55 * results.count)
+
+      info_urls = results
+                  .map { |record| record["info_url"] }
+                  .uniq
+                  .count { |text| text.to_s.match(%r{\Ahttps?://}) }
+      puts "Found #{info_urls} out of #{results.count} unique info_urls " \
+             "(#{(100.0 * info_urls / results.count).round(1)}%)"
+      expect(info_urls).to be > (0.7 * results.count) if info_urls != 1
+
+      VCR.use_cassette("#{authority}.info_urls") do
+        results.each do |record|
+          info_url = record["info_url"]
+          puts "Checking info_url #{info_url} #{info_urls > 1 ? ' has expected details' : ''} ..."
+          response = Net::HTTP.get_response(URI(info_url))
+
+          expect(response.code).to eq("200")
+          # If info_url is the same for all records, then it won't have details
+          break if info_urls == 1
+
+          expect(response.body).to include(record["council_reference"])
+          expect(response.body).to include(record["address"])
+          expect(response.body).to include(record["description"])
+        end
+      end
+    end
+
+    Scraper.selected_authorities.each do |authority|
+      it authority do
+        test_scrape(authority)
+      end
+    end
+  end
+end
+
+```

data/docs/example_scrape_with_fibers.rb

@@ -3,11 +3,11 @@
 # Example scrape method updated to use ScraperUtils::FibreScheduler
 
 def scrape(authorities, attempt)
-  ScraperUtils::FiberScheduler.reset!
+  ScraperUtils::Scheduler.reset!
   exceptions = {}
   authorities.each do |authority_label|
-    ScraperUtils::FiberScheduler.register_operation(authority_label) do
-      ScraperUtils::FiberScheduler.log(
+    ScraperUtils::Scheduler.register_operation(authority_label) do
+      ScraperUtils::LogUtils.log(
         "Collecting feed data for #{authority_label}, attempt: #{attempt}..."
       )
       ScraperUtils::DataQualityMonitor.start_authority(authority_label)
@@ -26,6 +26,6 @@ def scrape(authorities, attempt)
     end
     # end of register_operation block
   end
-  ScraperUtils::FiberScheduler.run_all
+  ScraperUtils::Scheduler.run_operations
   exceptions
 end

data/docs/fibers_and_threads.md

@@ -0,0 +1,72 @@
+Fibers and Threads
+==================
+
+This sequence diagram supplements the notes on the {ScraperUtils::Scheduler} class and is intended to help show
+the passing of messages and control between the fibers and threads.
+
+* To keep things simple I have only shown the Fibers and Threads and not all the other calls like to the
+  OperationRegistry to lookup the current operation, or OperationWorker etc.
+* There is ONE (global) response queue, which is monitored by the Scheduler.run_operations loop in the main Fiber
+* Each authority has ONE OperationWorker (not shown), which has ONE Fiber, ONE Thread, ONE request queue.
+* I use "◀─▶" to indicate a call and response, and "║" for which fiber / object is currently running.
+
+```text
+
+SCHEDULER (Main Fiber)      
+NxRegister-operation  RESPONSE.Q   
+   ║──creates────────◀─▶┐
+   ║                    │       FIBER (runs block passed to register_operation)
+   ║──creates──────────────────◀─▶┐         WORKER object and Registry
+   ║──registers(fiber)───────────────────────▶┐           REQUEST-Q
+   │                    │         │           ║──creates─◀─▶┐       THREAD
+   │                    │         │           ║──creates───────────◀─▶┐
+   ║◀─────────────────────────────────────────┘             ║◀──pop───║ ...[block waiting for request]
+   ║                    │         │           │             ║         │
+run_operations          │         │           │             ║         │
+   ║──pop(non block)─◀─▶│         │           │             ║         │ ...[no responses yet]
+   ║                    │         │           │             ║         │
+   ║───resumes-next─"can_resume"─────────────▶┐             ║         │
+   │                    │         │           ║             ║         │
+   │                    │         ║◀──resume──┘             ║         │ ...[first Resume passes true]
+   │                    │         ║           │             ║         │ ...[initialise scraper]
+```
+**REPEATS FROM HERE**  
+```text
+ SCHEDULER        RESPONSE.Q    FIBER       WORKER        REQUEST.Q THREAD
+   │                    │         ║──request─▶┐             ║         │
+   │                    │         │           ║──push req ─▶║         │
+   │                    │         ║◀──────────┘             ║──req───▶║
+   ║◀──yields control─(waiting)───┘           │             │         ║ 
+   ║                    │         │           │             │         ║ ...[Executes network I/O request]
+   ║                    │         │           │             │         ║
+   ║───other-resumes... │         │           │             │         ║ ...[Other Workers will be resumed 
+   ║                    │         │           │             │         ║     till most 99% are waiting on 
+   ║───lots of          │         │           │             │         ║     responses from their threads
+   ║    short sleeps    ║◀──pushes response───────────────────────────┘
+   ║                    ║         │           │             ║◀──pop───║ ...[block waiting for request]
+   ║──pop(response)──◀─▶║         │           │             ║         │
+   ║                    │         │           │             ║         │
+   ║──saves─response───────────────────────◀─▶│             ║         │
+   ║                    │         │           │             ║         │
+   ║───resumes-next─"can_resume"─────────────▶┐             ║         │
+   │                    │         │           ║             ║         │
+   │                    │         ║◀──resume──┘             ║         │ ...[Resume passes response]
+   │                    │         ║           │             ║         │ 
+   │                    │         ║           │             ║         │ ...[Process Response]
+```
+**REPEATS TO HERE** - WHEN FIBER FINISHES, instead it:  
+```text
+ SCHEDULER        RESPONSE.Q    FIBER         WORKER        REQUEST.Q THREAD  
+   │                    │         ║             │           ║         │ 
+   │                    │         ║─deregister─▶║           ║         │
+   │                    │         │             ║──close───▶║         │ 
+   │                    │         │             ║           ║──nil───▶┐ 
+   │                    │         │             ║           │         ║ ... [thread exists] 
+   │                    │         │             ║──join────────────◀─▶┘ 
+   │                    │         │             ║  ....... [worker removes 
+   │                    │         │             ║           itself from registry]                     
+   │                    │         ║◀──returns───┘                       
+   │◀──returns─nil────────────────┘                                    
+   │                    │                                             
+```
+When the last fiber finishes and the registry is empty, then the response queue is also removed

data/docs/getting_started.md

@@ -54,14 +54,14 @@ export DEBUG=1 # for basic, or 2 for verbose or 3 for tracing nearly everything
 
 ## Example Scraper Implementation
 
-Update your `scraper.rb` as per [example scraper](example_scraper.rb)
+Update your `scraper.rb` as per {file:example_scraper.rb example scraper}
 
-For more advanced implementations, see the [Interleaving Requests documentation](interleaving_requests.md).
+For more advanced implementations, see the {file:interleaving_requests.md Interleaving Requests documentation}.
 
 ## Logging Tables
 
 The following logging tables are created for use in monitoring failure patterns and debugging issues.
-Records are automaticaly cleared after 30 days.
+Records are automatically cleared after 30 days.
 
 The `ScraperUtils::LogUtils.log_scraping_run` call also logs the information to the `scrape_log` table.
 
@@ -69,6 +69,6 @@ The `ScraperUtils::LogUtils.save_summary_record` call also logs the information
 
 ## Next Steps
 
-- [Reducing Server Load](reducing_server_load.md)
-- [Mechanize Utilities](mechanize_utilities.md)
-- [Debugging](debugging.md)
+- {file:reducing_server_load.md Reducing Server Load}
+- {file:mechanize_utilities.md Mechanize Utilities}
+- {file:debugging.md Debugging}

data/docs/interleaving_requests.md

@@ -1,6 +1,6 @@
-# Interleaving Requests with FiberScheduler
+# Interleaving Requests with Scheduler
 
-The `ScraperUtils::FiberScheduler` provides a lightweight utility that:
+The `ScraperUtils::Scheduler` provides a lightweight utility that:
 
 * Works on other authorities while in the delay period for an authority's next request
 * Optimizes the total scraper run time
@@ -12,21 +12,22 @@ The `ScraperUtils::FiberScheduler` provides a lightweight utility that:
 ## Implementation
 
 To enable fiber scheduling, change your scrape method as per
-[example scrape with fibers](example_scrape_with_fibers.rb)
+{example_scrape_with_fibers.rb example scrape with fibers}
 
-## Logging with FiberScheduler
+## Logging with Scheduler
 
-Use `ScraperUtils::FiberScheduler.log` instead of `puts` when logging within the authority processing code.
+Use {ScraperUtils::LogUtils.log} instead of `puts` when logging within the authority processing code.
 This will prefix the output lines with the authority name, which is needed since the system will interleave the work and
 thus the output.
 
 ## Testing Considerations
 
-This uses `ScraperUtils::RandomizeUtils` for determining the order of operations. Remember to add the following line to
+This uses {ScraperUtils::RandomizeUtils} for determining the order of operations. Remember to add the following line to
 `spec/spec_helper.rb`:
 
 ```ruby
-ScraperUtils::RandomizeUtils.sequential = true
+ScraperUtils::RandomizeUtils.random = false
+ScraperUtils::Scheduler.max_workers = 1
 ```
 
-For full details, see the [FiberScheduler class documentation](https://rubydoc.info/gems/scraper_utils/ScraperUtils/FiberScheduler).
+For full details, see the {Scheduler}.

data/docs/mechanize_utilities.md

@@ -29,16 +29,16 @@ The agent returned is configured using Mechanize hooks to implement the desired
 ### Default Configuration
 
 By default, the Mechanize agent is configured with the following settings.
-As you can see, the defaults can be changed using env variables.
+As you can see, the defaults can be changed using env variables or via code.
 
 Note - compliant mode forces max_load to be set to a value no greater than 50.
 
 ```ruby
 ScraperUtils::MechanizeUtils::AgentConfig.configure do |config|
-  config.default_timeout = ENV.fetch('MORPH_TIMEOUT', 60).to_i # 60
+  config.default_timeout = ENV.fetch('MORPH_TIMEOUT', DEFAULT_TIMEOUT).to_i # 60
   config.default_compliant_mode = ENV.fetch('MORPH_NOT_COMPLIANT', nil).to_s.empty? # true
-  config.default_random_delay = ENV.fetch('MORPH_RANDOM_DELAY', 5).to_i # 5
-  config.default_max_load = ENV.fetch('MORPH_MAX_LOAD', 33.3).to_f # 33.3
+  config.default_random_delay = ENV.fetch('MORPH_RANDOM_DELAY', DEFAULT_RANDOM_DELAY).to_i # 0
+  config.default_max_load = ENV.fetch('MORPH_MAX_LOAD',DEFAULT_MAX_LOAD).to_f # 50.0
   config.default_disable_ssl_certificate_check = !ENV.fetch('MORPH_DISABLE_SSL_CHECK', nil).to_s.empty? # false
   config.default_australian_proxy = !ENV.fetch('MORPH_USE_PROXY', nil).to_s.empty? # false
   config.default_user_agent = ENV.fetch('MORPH_USER_AGENT', nil) # Uses Mechanize user agent