scraper_utils 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: db12d36e0d3be635eba2c00dbe149f4d177ddc5e538a08fcd9038a026feaee91
-  data.tar.gz: 8d2f140b7fff7e02d90df19ac196018f8719cd73d85067519ee3e931f679f619
+  metadata.gz: a2082c406ad96266f644fc1dfc588046aa43494e9e79b8fec38fe59252c09f06
+  data.tar.gz: 5100907cdcc8c55ddd59b25cf393d212f681b573ef874c5a6c65f748a8c852ec
 SHA512:
-  metadata.gz: 7138204493653a872aafcf4a1f8b78d8d5129c70d79a54d6ca10aa1440fc60362edc270522cc0d66c13a7694527a502d75d3dec36cb21f2240fceea85367eec4
-  data.tar.gz: 83dffaedd054ed40c7a269c4fd3db270892bc0fa20c4b7d1a904a075cb990bee51004ccb9c0cb86840d87a631655207301b8af1f7a5572f0893d9317a1b90aa5
+  metadata.gz: feabea23ac5b14f6b642769db303bcab954b22cfd2d95d7694af51afd661c73f1ff70a0c17b54557a730fe811275c126c0fdd4473e2ae6d2b83cfc495aa52bc6
+  data.tar.gz: d91154c0dbccfb4271fd4830a82316676c1ad5665773c30d212bcc1ab6178a09d4fee97b2b2a32480c32b8ee69da08fc9a78575460d49f8233fb91b74bf7df66

data/.gitignore

@@ -9,6 +9,9 @@
 /test/tmp/
 /test/version_tmp/
 
+# Ignore log files
+/log/
+
 # Temp files
 ,*
 *.bak

data/.rubocop.yml

@@ -1,12 +1,9 @@
+plugins:
+  - rubocop-rake
+  - rubocop-rspec
+
 AllCops:
-  Exclude:
-    - bin/*
-    # This is a temporary dumping ground for authority specific
-    # code that we're probably just initially copying across from
-    # other scrapers. So, we don't care about the formatting and style
-    # initially.
-    # TODO: Remove this once we've removed all the code from here
-    - lib/technology_one_scraper/authority/*
+  NewCops: enable
 
 # Bumping max line length to something a little more reasonable
 Layout/LineLength:

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+
+## 0.2.1 - 2025-02-28
+
+Fixed broken v0.2.0
+
+## 0.2.0 - 2025-02-28
+
+Added FiberScheduler, enabled complient mode with delays by default and simplified usage removing third retry without proxy
+
+## 0.1.0 - 2025-02-23
+
+First release for development

data/GUIDELINES.md

@@ -0,0 +1,75 @@
+# Project-Specific Guidelines
+
+These project specific guidelines supplement the general project instructions.
+Ask for clarification of any apparent conflicts with SPECS, IMPLEMENTATION or project instructions.
+
+## Error Handling Approaches
+
+Process each authority's site in issolation - problems with one authority are irrelevant to others.
+
+* we do a 2nd attempt of authorities with the same proxy settings
+* and a 3rd attemopt for those that failed with the proxy but with the proxy disabled
+
+Within a proxy distinguish between
+
+* Errors that are specific to that record
+    * only allow 5 such errors plus 10% of successfully processed records
+    * these could be regarded as not worth retrying (we currently do)
+* Any other exceptions stop the processing of that authorities site
+
+### Fail Fast on deeper calls
+
+- Raise exceptions early (when they are clearly detectable)
+- input validation according to scraper specs
+
+### Be forgiving on things that don't matter
+
+- not all sites have robots.txt, and not all robots.txt are well formatted therefor stop processing the file on obvious conflicts with the specs,
+but if the file is bad, just treat it as missing.
+
+- don't fuss over things we are not going to record.
+
+- we do detect maintenance pages because that is a helpful and simple clue that we wont find the data, and we can just wait till the site is back online
+
+## Type Checking
+
+### Partial Duck Typing
+- Focus on behavior over types internally (.robocop.yml should disable requiring everything to be typed)
+- Runtime validation of values
+- document public API though
+  - Use @params and @returns comments to document types for external uses of the public methods (rubymine will use thesefor checking)
+
+## Input Validation
+
+### Early Validation
+- Check all inputs at system boundaries
+- Fail on any invalid input
+
+## Testing Strategies
+
+* Avoid mocking unless really needed, instead
+  * instantiate a real object to use in the test
+  * use mocking facilities provided by the gem (eg Mechanize, Aws etc)
+  * use integration tests with WebMock for simple external sites or VCR for more complex.
+* Testing the integration all the way through is just as important as the specific algorithms
+* Consider using single responsibility classes / methods to make testing simpler but don't make things more complex just to be testable 
+* If necessary expose internal values as read only attributes as needed for testing, 
+  for example adding a read only attribute to mechanize agent instances with the values calculated internally
+
+### Behavior-Driven Development (BDD)
+- Focus on behavior specifications
+- User-centric scenarios
+- Best for: User-facing features
+
+## Documentation Approaches
+
+### Just-Enough Documentation
+- Focus on key decisions
+- Document non-obvious choices
+- Best for: Rapid development, internal tools
+
+## Logging Philosophy
+
+### Minimal Logging
+- Log only key events (key means down to adding a record)
+- Focus on errors

data/Gemfile

@@ -22,12 +22,15 @@ gem "sqlite3", platform && (platform == :heroku16 ? "~> 1.4.0" : "~> 1.6.3")
 gem "scraperwiki", git: "https://github.com/openaustralia/scraperwiki-ruby.git",
                    branch: "morph_defaults"
 
-# development and test test gems
+# development and test gems
 gem "rake", platform && (platform == :heroku16 ? "~> 12.3.3" : "~> 13.0")
 gem "rspec", platform && (platform == :heroku16 ? "~> 3.9.0" : "~> 3.12")
-gem "rubocop", platform && (platform == :heroku16 ? "~> 0.80.0" : "~> 1.57")
+gem "rubocop", platform && (platform == :heroku16 ? "~> 1.28.2" : "~> 1.73")
+gem "rubocop-rake", platform && (platform == :heroku16 ? "~> 0.6.0" : "~> 0.7")
+gem "rubocop-rspec", platform && (platform == :heroku16 ? "~> 2.10.0" : "~> 3.5")
 gem "simplecov", platform && (platform == :heroku16 ? "~> 0.18.0" : "~> 0.22.0")
-# gem "simplecov-console" listed in gemspec
+gem "simplecov-console"
+gem "terminal-table"
 gem "webmock", platform && (platform == :heroku16 ? "~> 3.14.0" : "~> 3.19.0")
 
 gemspec

data/IMPLEMENTATION.md

@@ -0,0 +1,33 @@
+IMPLEMENTATION
+==============
+
+Document decisions on how we are implementing the specs to be consistent and save time.
+Things we MUST do go in SPECS.
+Choices between a number of valid possibilities go here. 
+Once made, these choices should only be changed after careful consideration.
+
+ASK for clarification of any apparent conflicts with SPECS, GUIDELINES or project instructions.
+
+## Debugging
+
+Output debugging messages if ENV['DEBUG'] is set, for example:
+
+```ruby
+puts "Pre Connect request: #{request.inspect}" if ENV["DEBUG"]
+```
+
+## Robots.txt Handling
+
+- Used as a "good citizen" mechanism for respecting site preferences
+- Graceful fallback (to permitted) if robots.txt is unavailable or invalid
+- Match `/^User-agent:\s*ScraperUtils/i` for specific user agent
+  - If there is a line matching `/^Disallow:\s*\//` then we are disallowed
+  - Check for `/^Crawl-delay:\s*(\d[.0-9]*)/` to extract delay
+- If the no crawl-delay is found in that section, then check in the default `/^User-agent:\s*\*/` section
+- This is a deliberate significant simplification of the robots.txt specification in RFC 9309.
+
+## Method Organization
+
+- Externalize configuration to improve testability
+- Keep shared logic in the main class
+- Decisions / information specific to just one class, can be documented there, otherwise it belongs here

data/README.md

@@ -1,11 +1,53 @@
 ScraperUtils (Ruby)
 ===================
 
-Utilities to help make planningalerts scrapers, especially multis easier to develop, run and debug.
+Utilities to help make planningalerts scrapers, especially multis, easier to develop, run and debug.
 
-WARNING: This is still under development! Breaking changes may occur in version 0!
+WARNING: This is still under development! Breaking changes may occur in version 0.x!
 
-## Installation
+For Server Administrators
+-------------------------
+
+The ScraperUtils library is designed to be a respectful citizen of the web. If you're a server administrator and notice
+our scraper accessing your systems, here's what you should know:
+
+### How to Control Our Behavior
+
+Our scraper utilities respect the standard server **robots.txt** control mechanisms (by default).
+To control our access:
+
+- Add a section for our user agent: `User-agent: ScraperUtils` (default)
+- Set a crawl delay, eg: `Crawl-delay: 20`
+- If needed specify disallowed paths*: `Disallow: /private/`
+
+### Built-in Politeness Features
+
+Even without specific configuration, our scrapers will, by default:
+
+- **Identify themselves**: Our user agent clearly indicates who we are and provides a link to the project repository:
+  `Mozilla/5.0 (compatible; ScraperUtils/0.2.0 2025-02-22; +https://github.com/ianheggie-oaf/scraper_utils)`
+
+- **Limit server load**: We slow down our requests so we should never be a significant load to your server, let alone
+  overload it.
+  The slower your server is running, the longer the delay we add between requests to help.
+  In the default "compliant mode" this defaults to a max load of 20% and is capped at 33%.
+
+- **Add randomized delays**: We add random delays between requests to further reduce our impact on servers, which should
+  bring us
+- down to the load of a single industrious person.
+
+Extra utilities provided for scrapers to further reduce your server load:
+
+- **Interleave requests**: This spreads out the requests to your server rather than focusing on one scraper at a time.
+
+- **Intelligent Date Range selection**: This reduces server load by over 60% by a smarter choice of date range searches,
+  checking the recent 4 days each day and reducing down to checking each 3 days by the end of the 33-day mark. This
+  replaces the simplistic check of the last 30 days each day.
+
+Our goal is to access public planning information without negatively impacting your services.
+
+Installation
+------------
 
 Add these line to your application's Gemfile:
 
@@ -18,211 +60,133 @@ And then execute:
 
     $ bundle
 
-Or install it yourself for testing:
+Usage
+-----
+
+### Ruby Versions
 
-    $ gem install scraper_utils
+This gem is designed to be compatible the latest ruby supported by morph.io - other versions may work, but not tested:
 
-## Usage
+* ruby 3.2.2 - requires the `platform` file to contain `heroku_18` in the scraper
+* ruby 2.5.8 - `heroku_16` (the default)
 
 ### Environment variables
 
-Optionally filter authorities via environment variable in morph > scraper > settings or 
+#### `MORPH_AUSTRALIAN_PROXY`
+
+On morph.io set the environment variable `MORPH_AUSTRALIAN_PROXY` to
+`http://morph:password@au.proxy.oaf.org.au:8888`
+replacing password with the real password.
+Alternatively enter your own AUSTRALIAN proxy details when testing.
+
+#### `MORPH_EXPECT_BAD`
+
+To avoid morph complaining about sites that are known to be bad,
+but you want them to keep being tested, list them on `MORPH_EXPECT_BAD`, for example:
+
+#### `MORPH_AUTHORITIES`
+
+Optionally filter authorities for multi authority scrapers
+via environment variable in morph > scraper > settings or
 in your dev environment:
 
 ```bash
 export MORPH_AUTHORITIES=noosa,wagga
 ```
 
-### Example updated `scraper.rb` file
+#### `DEBUG`
 
-Update your `scraper.rb` as per the following example:
+Optionally enable verbose debugging messages when developing:
 
-```ruby
-#!/usr/bin/env ruby
-# frozen_string_literal: true
+```bash
+export DEBUG=1
+```
 
-$LOAD_PATH << "./lib"
+### Extra Mechanize options
 
-require "scraper_utils"
-require "technology_one_scraper"
-
-# Main Scraper class
-class Scraper
-  AUTHORITIES = TechnologyOneScraper::AUTHORITIES
-
-  def self.scrape(authorities, attempt)
-    results = {}
-    authorities.each do |authority_label|
-      these_results = results[authority_label] = {}
-      begin
-        records_scraped = 0
-        unprocessable_records = 0
-        # Allow 5 + 10% unprocessable records
-        too_many_unprocessable = -5.0
-        use_proxy = AUTHORITIES[authority_label][:australian_proxy] && ScraperUtils.australian_proxy
-        next if attempt > 2 && !use_proxy
-
-        puts "",
-             "Collecting feed data for #{authority_label}, attempt: #{attempt}" \
-               "#{use_proxy ? ' (via proxy)' : ''} ..."
-        # Change scrape to accept a use_proxy flag and return an unprocessable flag
-        # it should rescue ScraperUtils::UnprocessableRecord thrown deeper in the scraping code and
-        # set unprocessable
-        TechnologyOneScraper.scrape(use_proxy, authority_label) do |record, unprocessable|
-          unless unprocessable
-            begin
-              record["authority_label"] = authority_label.to_s
-              ScraperUtils::DbUtils.save_record(record)
-            rescue ScraperUtils::UnprocessableRecord => e
-              # validation error
-              unprocessable = true
-              these_results[:error] = e
-            end
-          end
-          if unprocessable
-            unprocessable_records += 1
-            these_results[:unprocessable_records] = unprocessable_records
-            too_many_unprocessable += 1
-            raise "Too many unprocessable records" if too_many_unprocessable.positive?
-          else
-            records_scraped += 1
-            these_results[:records_scraped] = records_scraped
-            too_many_unprocessable -= 0.1
-          end
-        end
-      rescue StandardError => e
-        warn "#{authority_label}: ERROR: #{e}"
-        warn e.backtrace || "No backtrace available"
-        these_results[:error] = e
-      end
-    end
-    results
-  end
+Add `client_options` to your AUTHORITIES configuration and move any of the following settings into it:
 
-  def self.selected_authorities
-    ScraperUtils::AuthorityUtils.selected_authorities(AUTHORITIES.keys)
-  end
+* `timeout: Integer` - Timeout for agent connections in case the server is slower than normal
+* `australian_proxy: true` - Use the proxy url in the `MORPH_AUSTRALIAN_PROXY` env variable if the site is geo-locked
+* `disable_ssl_certificate_check: true` - Disabled SSL verification for old / incorrect certificates
 
-  def self.run(authorities)
-    puts "Scraping authorities: #{authorities.join(', ')}"
-    start_time = Time.now
-    results = scrape(authorities, 1)
-    ScraperUtils::LogUtils.log_scraping_run(
-      start_time,
-      1,
-      authorities,
-      results
-    )
-
-    retry_errors = results.select do |_auth, result|
-      result[:error] && !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-    end.keys
-
-    unless retry_errors.empty?
-      puts "",
-           "***************************************************"
-      puts "Now retrying authorities which earlier had failures"
-      puts retry_errors.join(", ").to_s
-      puts "***************************************************"
-
-      start_retry = Time.now
-      retry_results = scrape(retry_errors, 2)
-      ScraperUtils::LogUtils.log_scraping_run(
-        start_retry,
-        2,
-        retry_errors,
-        retry_results
-      )
-
-      retry_results.each do |auth, result|
-        unless result[:error] && !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-          results[auth] = result
-        end
-      end.keys
-      retry_no_proxy = retry_results.select do |_auth, result|
-        result[:used_proxy] && result[:error] &&
-          !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-      end.keys
-
-      unless retry_no_proxy.empty?
-        puts "",
-             "*****************************************************************"
-        puts "Now retrying authorities which earlier had failures without proxy"
-        puts retry_no_proxy.join(", ").to_s
-        puts "*****************************************************************"
-
-        start_retry = Time.now
-        second_retry_results = scrape(retry_no_proxy, 3)
-        ScraperUtils::LogUtils.log_scraping_run(
-          start_retry,
-          3,
-          retry_no_proxy,
-          second_retry_results
-        )
-        second_retry_results.each do |auth, result|
-          unless result[:error] && !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-            results[auth] = result
-          end
-        end.keys
-      end
-    end
-
-    # Report on results, raising errors for unexpected conditions
-    ScraperUtils::LogUtils.report_on_results(authorities, results)
-  end
+See the documentation on `ScraperUtils::MechanizeUtils::AgentConfig` for more options
+
+Then adjust your code to accept `client_options` and pass then through to:
+`ScraperUtils::MechanizeUtils.mechanize_agent(client_options || {})`
+to receive a `Mechanize::Agent` configured accordingly.
+
+The agent returned is configured using Mechanize hooks to implement the desired delays automatically.
+
+### Default Configuration
+
+By default, the Mechanize agent is configured with the following settings.
+As you can see, the defaults can be changed using env variables.
+
+Note - compliant mode forces max_load to be set to a value no greater than 33.
+PLEASE don't use our user agent string with a max_load higher than 33!
+
+```ruby
+ScraperUtils::MechanizeUtils::AgentConfig.configure do |config|
+  config.default_timeout = ENV.fetch('MORPH_TIMEOUT', 60).to_i # 60
+  config.default_compliant_mode = ENV.fetch('MORPH_NOT_COMPLIANT', nil).to_s.empty? # true
+  config.default_random_delay = ENV.fetch('MORPH_RANDOM_DELAY', 15).to_i # 15
+  config.default_max_load = ENV.fetch('MORPH_MAX_LOAD', 20.0).to_f # 20
+  config.default_disable_ssl_certificate_check = !ENV.fetch('MORPH_DISABLE_SSL_CHECK', nil).to_s.empty? # false
+  config.default_australian_proxy = !ENV.fetch('MORPH_USE_PROXY', nil).to_s.empty? # false
+  config.default_user_agent = ENV.fetch('MORPH_USER_AGENT', nil) # Uses Mechanize user agent
 end
+```
+
+You can modify these global defaults before creating any Mechanize agents. These settings will be used for all Mechanize
+agents created by `ScraperUtils::MechanizeUtils.mechanize_agent` unless overridden by passing parameters to that method.
 
-if __FILE__ == $PROGRAM_NAME
-  # Default to list of authorities we can't or won't fix in code, explain why
-  # wagga: url redirects and reports Application error, main site says to use NSW Planning Portal from 1 July 2021
-  #        which doesn't list any DA's for wagga wagga!
+To speed up testing, set the following in `spec_helper.rb`:
 
-  ENV["MORPH_EXPECT_BAD"] ||= "wagga"
-  Scraper.run(Scraper.selected_authorities)
+```ruby
+ScraperUtils::MechanizeUtils::AgentConfig.configure do |config|
+  config.default_random_delay = nil
+  config.default_max_load = 33
 end
 ```
 
-Then deeper in your code update:
+### Example updated `scraper.rb` file
 
-* Change scrape to accept a `use_proxy` flag and return an `unprocessable` flag
-* it should rescue ScraperUtils::UnprocessableRecord thrown deeper in the scraping code and
-  set and yield unprocessable eg: `TechnologyOneScraper.scrape(use_proxy, authority_label) do |record, unprocessable|`
+Update your `scraper.rb` as per the [example scraper](docs/example_scraper.rb).
+
+Your code should raise ScraperUtils::UnprocessableRecord when there is a problem with the data presented on a page for a
+record.
+Then just before you would normally yield a record for saving, rescue that exception and:
+
+* Call `ScraperUtils::DataQualityMonitor.log_unprocessable_record(e, record)`
+* NOT yield the record for saving
+
+In your code update where create a mechanize agent (often `YourScraper.scrape_period`) and the `AUTHORITIES` hash
+to move Mechanize agent options (like `australian_proxy` and `timeout`) to a hash under a new key: `client_options`.
+For example:
 
 ```ruby
 require "scraper_utils"
 #...
-module TechnologyOneScraper
-  # Note the extra parameter: use_proxy
-  def self.scrape(use_proxy, authority)
-    raise "Unexpected authority: #{authority}" unless AUTHORITIES.key?(authority)
-
-    scrape_period(use_proxy, AUTHORITIES[authority]) do |record, unprocessable|
-      yield record, unprocessable
-    end
-  end
-  
-  # ... rest of code ...
+module YourScraper
+  # ... some code ...
 
-  # Note the extra parameters: use_proxy and timeout
-  def self.scrape_period(use_proxy,
-    url:, period:, webguest: "P1.WEBGUEST", disable_ssl_certificate_check: false,
-    australian_proxy: false, timeout: nil
+  # Note the extra parameter: client_options
+  def self.scrape_period(url:, period:, webguest: "P1.WEBGUEST",
+                         client_options: {}
   )
-    agent = ScraperUtils::MechanizeUtils.mechanize_agent(use_proxy: use_proxy, timeout: timeout)
-    agent.verify_mode = OpenSSL::SSL::VERIFY_NONE if disable_ssl_certificate_check
-    
-    # ... rest of code ...
-
-    # Update yield to return unprocessable as well as record
+    agent = ScraperUtils::MechanizeUtils.mechanize_agent(**client_options)
 
+    # ... rest of code ...
   end
+
   # ... rest of code ...
 end
 ```
 
 ### Debugging Techniques
 
-The following code will print dbugging info if you set:
+The following code will cause debugging info to be output:
 
 ```bash
 export DEBUG=1
@@ -235,8 +199,8 @@ require 'scraper_utils'
 
 # Debug an HTTP request
 ScraperUtils::DebugUtils.debug_request(
-  "GET", 
-  "https://example.com/planning-apps", 
+  "GET",
+  "https://example.com/planning-apps",
   parameters: { year: 2023 },
   headers: { "Accept" => "application/json" }
 )
@@ -248,7 +212,85 @@ ScraperUtils::DebugUtils.debug_page(page, "Checking search results page")
 ScraperUtils::DebugUtils.debug_selector(page, '.results-table', "Looking for development applications")
 ```
 
-## Development
+Interleaving Requests
+---------------------
+
+The `ScraperUtils::FiberScheduler` provides a lightweight utility that:
+
+* works on the other authorities whilst in the delay period for an authorities next request
+* thus optimizing the total scraper run time
+* allows you to increase the random delay for authorities without undue effect on total run time
+* For the curious, it uses [ruby fibers](https://ruby-doc.org/core-2.5.8/Fiber.html) rather than threads as that is
+  a simpler system and thus easier to get right, understand and debug!
+* Cycles around the authorities when compliant_mode, max_load and random_delay are disabled
+
+To enable change the scrape method to be like [example scrape method using fibers](docs/example_scrape_with_fibers.rb)
+
+And use `ScraperUtils::FiberScheduler.log` instead of `puts` when logging within the authority processing code.
+This will prefix the output lines with the authority name, which is needed since the system will interleave the work and
+thus the output.
+
+This uses `ScraperUtils::RandomizeUtils` as described below. Remember to add the recommended line to
+`spec/spec_heper.rb`.
+
+Intelligent Date Range Selection
+--------------------------------                                                                                                                                                                                   
+
+To further reduce server load and speed up scrapers, we provide an intelligent date range selection mechanism
+that can reduce server requests by 60% without significantly impacting delay in picking up changes.
+
+The `ScraperUtils::DateRangeUtils#calculate_date_ranges` method provides a smart approach to searching historical
+records:
+
+- Always checks the most recent 4 days daily (configurable)
+- Progressively reduces search frequency for older records
+- Uses a Fibonacci-like progression to create natural, efficient search intervals
+- Configurable `max_period` (default is 3 days)
+- merges adjacent search ranges and handles the changeover in search frequency by extending some searches
+
+Example usage in your scraper:
+
+ ```ruby                                                                                                                                                                                                            
+date_ranges = ScraperUtils::DateRangeUtils.new.calculate_date_ranges
+date_ranges.each do |from_date, to_date, _debugging_comment|
+  # Adjust your normal search code to use for this date range                                                                                                                                                                             
+  your_search_records(from_date: from_date, to_date: to_date) do |record|
+    # process as normal
+  end
+end
+```
+
+Typical server load reductions:
+
+* Max period 2 days : ~42% of the 33 days selected
+* Max period 3 days : ~37% of the 33 days selected (default)
+* Max period 5 days : ~35% (or ~31% when days = 45)
+
+See the class documentation for customizing defaults and passing options.
+
+Randomizing Requests
+--------------------
+
+Pass a `Collection` or `Array` to `ScraperUtils::RandomizeUtils.randomize_order` to randomize it in production, but
+receive in as is when testing.
+
+Use this with the list of records scraped from an index to randomise any requests for further information to be less Bot like.
+
+### Spec setup
+
+You should enforce sequential mode when testing by adding the following code to `spec/spec_helper.rb` :
+
+```
+ScraperUtils::RandomizeUtils.sequential = true
+```
+
+Note:
+
+* You can also force sequential mode by setting the env variable `MORPH_PROCESS_SEQUENTIALLY` to `1` (any non blank)
+* testing using VCR requires sequential mode
+
+Development
+-----------
 
 After checking out the repo, run `bin/setup` to install dependencies.
 Then, run `rake test` to run the tests.
@@ -259,13 +301,20 @@ To install this gem onto your local machine, run `bundle exec rake install`.
 
 To release a new version, update the version number in `version.rb`, and
 then run `bundle exec rake release`,
-which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+which will create a git tag for the version, push git commits and tags, and push the `.gem` file
+to [rubygems.org](https://rubygems.org).
+
+NOTE: You need to use ruby 3.2.2 instead of 2.5.8 to release to OTP protected accounts.
+
+Contributing
+------------
 
-## Contributing
+Bug reports and pull requests with working tests are welcome on [GitHub](https://github.com/ianheggie-oaf/scraper_utils)
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/ianheggie-oaf/scraper_utils
+CHANGELOG.md is maintained by the author aiming to follow https://github.com/vweevers/common-changelog
 
-## License
+License
+-------
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).