scraper_utils 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: db12d36e0d3be635eba2c00dbe149f4d177ddc5e538a08fcd9038a026feaee91
-  data.tar.gz: 8d2f140b7fff7e02d90df19ac196018f8719cd73d85067519ee3e931f679f619
+  metadata.gz: b4291b6994419c04851935fe4aa4e047eb4069cab3fecf451bf65f8e91acb48d
+  data.tar.gz: 2e3a657ce230f9c6bc9defe042cf7babb9e52e2130d32f0ec8312571f5dcb26a
 SHA512:
-  metadata.gz: 7138204493653a872aafcf4a1f8b78d8d5129c70d79a54d6ca10aa1440fc60362edc270522cc0d66c13a7694527a502d75d3dec36cb21f2240fceea85367eec4
-  data.tar.gz: 83dffaedd054ed40c7a269c4fd3db270892bc0fa20c4b7d1a904a075cb990bee51004ccb9c0cb86840d87a631655207301b8af1f7a5572f0893d9317a1b90aa5
+  metadata.gz: 51c29aea77f43a8c7de8e874a2601b4e0b9e9c36ae512180f10dcd182b2ecc899cc08944face686e0f993b02338975672d9eaf06d8a0185ee222cfc263993244
+  data.tar.gz: ec63009a4f10677a8e9500b5ca15c68b432081fa995d5bee2eda2b2cff88cdb7090595be7ad64a0dfeacf09decbc292313b47f9f8af795a3b95c718c77f59339

data/.rubocop.yml

@@ -1,12 +1,5 @@
 AllCops:
-  Exclude:
-    - bin/*
-    # This is a temporary dumping ground for authority specific
-    # code that we're probably just initially copying across from
-    # other scrapers. So, we don't care about the formatting and style
-    # initially.
-    # TODO: Remove this once we've removed all the code from here
-    - lib/technology_one_scraper/authority/*
+  NewCops: enable
 
 # Bumping max line length to something a little more reasonable
 Layout/LineLength:

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog
+
+## 0.1.0 - 2025-02-23
+
+`First release for development`

data/GUIDELINES.md

@@ -0,0 +1,75 @@
+# Project-Specific Guidelines
+
+These project specific guidelines supplement the general project instructions.
+Ask for clarification of any apparent conflicts with SPECS, IMPLEMENTATION or project instructions.
+
+## Error Handling Approaches
+
+Process each authority's site in issolation - problems with one authority are irrelevant to others.
+
+* we do a 2nd attempt of authorities with the same proxy settings
+* and a 3rd attemopt for those that failed with the proxy but with the proxy disabled
+
+Within a proxy distinguish between
+
+* Errors that are specific to that record
+    * only allow 5 such errors plus 10% of successfully processed records
+    * these could be regarded as not worth retrying (we currently do)
+* Any other exceptions stop the processing of that authorities site
+
+### Fail Fast on deeper calls
+
+- Raise exceptions early (when they are clearly detectable)
+- input validation according to scraper specs
+
+### Be forgiving on things that don't matter
+
+- not all sites have robots.txt, and not all robots.txt are well formatted therefor stop processing the file on obvious conflicts with the specs,
+but if the file is bad, just treat it as missing.
+
+- don't fuss over things we are not going to record.
+
+- we do detect maintenance pages because that is a helpful and simple clue that we wont find the data, and we can just wait till the site is back online
+
+## Type Checking
+
+### Partial Duck Typing
+- Focus on behavior over types internally (.robocop.yml should disable requiring everything to be typed)
+- Runtime validation of values
+- document public API though
+  - Use @params and @returns comments to document types for external uses of the public methods (rubymine will use thesefor checking)
+
+## Input Validation
+
+### Early Validation
+- Check all inputs at system boundaries
+- Fail on any invalid input
+
+## Testing Strategies
+
+* Avoid mocking unless really needed, instead
+  * instantiate a real object to use in the test
+  * use mocking facilities provided by the gem (eg Mechanize, Aws etc)
+  * use integration tests with WebMock for simple external sites or VCR for more complex.
+* Testing the integration all the way through is just as important as the specific algorithms
+* Consider using single responsibility classes / methods to make testing simpler but don't make things more complex just to be testable 
+* If necessary expose internal values as read only attributes as needed for testing, 
+  for example adding a read only attribute to mechanize agent instances with the values calculated internally
+
+### Behavior-Driven Development (BDD)
+- Focus on behavior specifications
+- User-centric scenarios
+- Best for: User-facing features
+
+## Documentation Approaches
+
+### Just-Enough Documentation
+- Focus on key decisions
+- Document non-obvious choices
+- Best for: Rapid development, internal tools
+
+## Logging Philosophy
+
+### Minimal Logging
+- Log only key events (key means down to adding a record)
+- Focus on errors

data/Gemfile

@@ -27,7 +27,7 @@ gem "rake", platform && (platform == :heroku16 ? "~> 12.3.3" : "~> 13.0")
 gem "rspec", platform && (platform == :heroku16 ? "~> 3.9.0" : "~> 3.12")
 gem "rubocop", platform && (platform == :heroku16 ? "~> 0.80.0" : "~> 1.57")
 gem "simplecov", platform && (platform == :heroku16 ? "~> 0.18.0" : "~> 0.22.0")
-# gem "simplecov-console" listed in gemspec
+gem "simplecov-console"
 gem "webmock", platform && (platform == :heroku16 ? "~> 3.14.0" : "~> 3.19.0")
 
 gemspec

data/IMPLEMENTATION.md

@@ -0,0 +1,33 @@
+IMPLEMENTATION
+==============
+
+Document decisions on how we are implementing the specs to be consistent and save time.
+Things we MUST do go in SPECS.
+Choices between a number of valid possibilities go here. 
+Once made, these choices should only be changed after careful consideration.
+
+ASK for clarification of any apparent conflicts with SPECS, GUIDELINES or project instructions.
+
+## Debugging
+
+Output debugging messages if ENV['DEBUG'] is set, for example:
+
+```ruby
+puts "Pre Connect request: #{request.inspect}" if ENV["DEBUG"]
+```
+
+## Robots.txt Handling
+
+- Used as a "good citizen" mechanism for respecting site preferences
+- Graceful fallback (to permitted) if robots.txt is unavailable or invalid
+- Match `/^User-agent:\s*ScraperUtils/i` for specific user agent
+  - If there is a line matching `/^Disallow:\s*\//` then we are disallowed
+  - Check for `/^Crawl-delay:\s*(\d[.0-9]*)/` to extract delay
+- If the no crawl-delay is found in that section, then check in the default `/^User-agent:\s*\*/` section
+- This is a deliberate significant simplification of the robots.txt specification in RFC 9309.
+
+## Method Organization
+
+- Externalize configuration to improve testability
+- Keep shared logic in the main class
+- Decisions / information specific to just one class, can be documented there, otherwise it belongs here

data/README.md

@@ -1,11 +1,43 @@
 ScraperUtils (Ruby)
 ===================
 
-Utilities to help make planningalerts scrapers, especially multis easier to develop, run and debug.
+Utilities to help make planningalerts scrapers, especially multis, easier to develop, run and debug.
 
-WARNING: This is still under development! Breaking changes may occur in version 0!
+WARNING: This is still under development! Breaking changes may occur in version 0.x!
 
-## Installation
+For Server Administrators
+-------------------------
+
+The ScraperUtils library is designed to be a respectful citizen of the web. If you're a server administrator and notice
+our scraper accessing your systems, here's what you should know:
+
+### How to Control Our Behavior
+
+Our scraper utilities respect the standard server **robots.txt** control mechanisms (by default). To control our access:
+
+- Add a section for our user agent: `User-agent: ScraperUtils` (default)
+- Set a crawl delay: `Crawl-delay: 5`
+- If needed specify disallowed paths: `Disallow: /private/`
+
+### Built-in Politeness Features
+
+Even without specific configuration, our scrapers will, by default:
+
+- **Identify themselves**: Our user agent clearly indicates who we are and provides a link to the project repository:
+  `Mozilla/5.0 (compatible; ScraperUtils/0.2.0 2025-02-22; +https://github.com/ianheggie-oaf/scraper_utils)`
+
+- **Limit server load**: We introduce delays to avoid undue load on your server's by default based on your response
+  time.
+  The slower your server is running, the longer the delay we add between requests to help you.
+  In the default "compliant mode" this defaults to 20% and custom settings are capped at 33% maximum.
+
+- **Add randomized delays**: We add random delays between requests to avoid creating regular traffic patterns that might
+  impact server performance (enabled by default).
+
+Our goal is to access public planning information without negatively impacting your services.
+
+Installation
+------------
 
 Add these line to your application's Gemfile:
 
@@ -22,20 +54,85 @@ Or install it yourself for testing:
 
     $ gem install scraper_utils
 
-## Usage
+Usage
+-----
+
+### Ruby Versions
+
+This gem is designed to be compatible the latest ruby supported by morph.io - other versions may work, but not tested:
+
+* ruby 3.2.2 - requires the `platform` file to contain `heroku_18` in the scraper
+* ruby 2.5.8 - `heroku_16` (the default)
 
 ### Environment variables
 
-Optionally filter authorities via environment variable in morph > scraper > settings or 
+#### `MORPH_AUSTRALIAN_PROXY`
+
+On morph.io set the environment variable `MORPH_AUSTRALIAN_PROXY` to
+`http://morph:password@au.proxy.oaf.org.au:8888`
+replacing password with the real password.
+Alternatively enter your own AUSTRALIAN proxy details when testing.
+
+#### `MORPH_EXPECT_BAD`
+
+To avoid morph complaining about sites that are known to be bad,
+but you want them to keep being tested, list them on `MORPH_EXPECT_BAD`, for example:
+
+#### `MORPH_AUTHORITIES`
+
+Optionally filter authorities for multi authority scrapers
+via environment variable in morph > scraper > settings or
 in your dev environment:
 
 ```bash
 export MORPH_AUTHORITIES=noosa,wagga
 ```
 
+#### `DEBUG`
+
+Optionally enable verbose debugging messages when developing:
+
+```bash
+export DEBUG=1
+```
+
+### Extra Mechanize options
+
+Add `client_options` to your AUTHORITIES configuration and move any of the following settings into it:
+
+* `timeout: Integer` - Timeout for agent connections in case the server is slower than normal
+* `australian_proxy: true` - Use the MORPH_AUSTRALIAN_PROXY as proxy url if the site is geo-locked
+* `disable_ssl_certificate_check: true` - Disabled SSL verification for old / incorrect certificates
+
+See the documentation on `ScraperUtils::MechanizeUtils::AgentConfig` for more options
+
+Then adjust your code to accept client_options and pass then through to:
+`ScraperUtils::MechanizeUtils.mechanize_agent(client_options || {})`
+to receive a `Mechanize::Agent` configured accordingly.
+
+The agent returned is configured using Mechanize hooks to implement the desired delays automatically.
+
+### Default Configuration
+
+By default, the Mechanize agent is configured with the following settings.
+
+```ruby
+ScraperUtils::MechanizeUtils::AgentConfig.configure do |config|
+  config.default_timeout = 60
+  config.default_compliant_mode = true
+  config.default_random_delay = 3
+  config.default_max_load = 20 # percentage
+  config.default_disable_ssl_certificate_check = false
+  config.default_australian_proxy = false
+end
+```
+
+You can modify these global defaults before creating any Mechanize agents. These settings will be used for all Mechanize
+agents created by `ScraperUtils::MechanizeUtils.mechanize_agent` unless overridden by passing parameters to that method.
+
 ### Example updated `scraper.rb` file
 
-Update your `scraper.rb` as per the following example:
+Update your `scraper.rb` as per the following example for basic utilities:
 
 ```ruby
 #!/usr/bin/env ruby
@@ -48,55 +145,42 @@ require "technology_one_scraper"
 
 # Main Scraper class
 class Scraper
-  AUTHORITIES = TechnologyOneScraper::AUTHORITIES
+  AUTHORITIES = YourScraper::AUTHORITIES
 
-  def self.scrape(authorities, attempt)
-    results = {}
+  # ADD: attempt argument
+  def scrape(authorities, attempt)
+    exceptions = {}
+    # ADD: Report attempt number
     authorities.each do |authority_label|
-      these_results = results[authority_label] = {}
+      puts "\nCollecting feed data for #{authority_label}, attempt: #{attempt}..."
+
       begin
-        records_scraped = 0
-        unprocessable_records = 0
-        # Allow 5 + 10% unprocessable records
-        too_many_unprocessable = -5.0
-        use_proxy = AUTHORITIES[authority_label][:australian_proxy] && ScraperUtils.australian_proxy
-        next if attempt > 2 && !use_proxy
-
-        puts "",
-             "Collecting feed data for #{authority_label}, attempt: #{attempt}" \
-               "#{use_proxy ? ' (via proxy)' : ''} ..."
-        # Change scrape to accept a use_proxy flag and return an unprocessable flag
-        # it should rescue ScraperUtils::UnprocessableRecord thrown deeper in the scraping code and
-        # set unprocessable
-        TechnologyOneScraper.scrape(use_proxy, authority_label) do |record, unprocessable|
-          unless unprocessable
-            begin
-              record["authority_label"] = authority_label.to_s
-              ScraperUtils::DbUtils.save_record(record)
-            rescue ScraperUtils::UnprocessableRecord => e
-              # validation error
-              unprocessable = true
-              these_results[:error] = e
-            end
-          end
-          if unprocessable
-            unprocessable_records += 1
-            these_results[:unprocessable_records] = unprocessable_records
-            too_many_unprocessable += 1
-            raise "Too many unprocessable records" if too_many_unprocessable.positive?
-          else
-            records_scraped += 1
-            these_results[:records_scraped] = records_scraped
-            too_many_unprocessable -= 0.1
+        # REPLACE:
+        # YourScraper.scrape(authority_label) do |record|
+        #   record["authority_label"] = authority_label.to_s
+        #   YourScraper.log(record)
+        #   ScraperWiki.save_sqlite(%w[authority_label council_reference], record)
+        # end
+        # WITH:
+        ScraperUtils::DataQualityMonitor.start_authority(authority_label)
+        YourScraper.scrape(authority_label) do |record|
+          begin
+            record["authority_label"] = authority_label.to_s
+            ScraperUtils::DbUtils.save_record(record)
+          rescue ScraperUtils::UnprocessableRecord => e
+            ScraperUtils::DataQualityMonitor.log_unprocessable_record(e, record)
+            exceptions[authority_label] = e
           end
         end
-      rescue StandardError => e
-        warn "#{authority_label}: ERROR: #{e}"
-        warn e.backtrace || "No backtrace available"
-        these_results[:error] = e
+        # END OF REPLACE
       end
+    rescue StandardError => e
+      warn "#{authority_label}: ERROR: #{e}"
+      warn e.backtrace
+      exceptions[authority_label] = e
     end
-    results
+
+    exceptions
   end
 
   def self.selected_authorities
@@ -106,123 +190,79 @@ class Scraper
   def self.run(authorities)
     puts "Scraping authorities: #{authorities.join(', ')}"
     start_time = Time.now
-    results = scrape(authorities, 1)
+    exceptions = scrape(authorities, 1)
+    # Set start_time and attempt to the call above and log run below
     ScraperUtils::LogUtils.log_scraping_run(
       start_time,
       1,
       authorities,
-      results
+      exceptions
     )
 
-    retry_errors = results.select do |_auth, result|
-      result[:error] && !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-    end.keys
-
-    unless retry_errors.empty?
-      puts "",
-           "***************************************************"
+    unless exceptions.empty?
+      puts "\n***************************************************"
       puts "Now retrying authorities which earlier had failures"
-      puts retry_errors.join(", ").to_s
+      puts exceptions.keys.join(", ").to_s
       puts "***************************************************"
 
-      start_retry = Time.now
-      retry_results = scrape(retry_errors, 2)
+      start_time = Time.now
+      exceptions = scrape(exceptions.keys, 2)
+      # Set start_time and attempt to the call above and log run below
       ScraperUtils::LogUtils.log_scraping_run(
-        start_retry,
+        start_time,
         2,
-        retry_errors,
-        retry_results
+        authorities,
+        exceptions
       )
-
-      retry_results.each do |auth, result|
-        unless result[:error] && !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-          results[auth] = result
-        end
-      end.keys
-      retry_no_proxy = retry_results.select do |_auth, result|
-        result[:used_proxy] && result[:error] &&
-          !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-      end.keys
-
-      unless retry_no_proxy.empty?
-        puts "",
-             "*****************************************************************"
-        puts "Now retrying authorities which earlier had failures without proxy"
-        puts retry_no_proxy.join(", ").to_s
-        puts "*****************************************************************"
-
-        start_retry = Time.now
-        second_retry_results = scrape(retry_no_proxy, 3)
-        ScraperUtils::LogUtils.log_scraping_run(
-          start_retry,
-          3,
-          retry_no_proxy,
-          second_retry_results
-        )
-        second_retry_results.each do |auth, result|
-          unless result[:error] && !result[:error].is_a?(ScraperUtils::UnprocessableRecord)
-            results[auth] = result
-          end
-        end.keys
-      end
     end
 
     # Report on results, raising errors for unexpected conditions
-    ScraperUtils::LogUtils.report_on_results(authorities, results)
+    ScraperUtils::LogUtils.report_on_results(authorities, exceptions)
   end
 end
 
 if __FILE__ == $PROGRAM_NAME
   # Default to list of authorities we can't or won't fix in code, explain why
-  # wagga: url redirects and reports Application error, main site says to use NSW Planning Portal from 1 July 2021
-  #        which doesn't list any DA's for wagga wagga!
+  # wagga: url redirects and then reports Application error
 
   ENV["MORPH_EXPECT_BAD"] ||= "wagga"
   Scraper.run(Scraper.selected_authorities)
 end
 ```
 
-Then deeper in your code update:
+Your code should raise ScraperUtils::UnprocessableRecord when there is a problem with the data presented on a page for a
+record.
+Then just before you would normally yield a record for saving, rescue that exception and:
+
+* Call ScraperUtils::DataQualityMonitor.log_unprocessable_record(e, record)
+* NOT yield the record for saving
 
-* Change scrape to accept a `use_proxy` flag and return an `unprocessable` flag
-* it should rescue ScraperUtils::UnprocessableRecord thrown deeper in the scraping code and
-  set and yield unprocessable eg: `TechnologyOneScraper.scrape(use_proxy, authority_label) do |record, unprocessable|`
+In your code update where create a mechanize agent (often `YourScraper.scrape_period`) and the `AUTHORITIES` hash
+to move mechanize_agent options (like `australian_proxy` and `timeout`) to a hash under a new key: `client_options`.
+For example:
 
 ```ruby
 require "scraper_utils"
 #...
-module TechnologyOneScraper
-  # Note the extra parameter: use_proxy
-  def self.scrape(use_proxy, authority)
-    raise "Unexpected authority: #{authority}" unless AUTHORITIES.key?(authority)
+module YourScraper
+  # ... some code ...
 
-    scrape_period(use_proxy, AUTHORITIES[authority]) do |record, unprocessable|
-      yield record, unprocessable
-    end
-  end
-  
-  # ... rest of code ...
-
-  # Note the extra parameters: use_proxy and timeout
-  def self.scrape_period(use_proxy,
-    url:, period:, webguest: "P1.WEBGUEST", disable_ssl_certificate_check: false,
-    australian_proxy: false, timeout: nil
+  # Note the extra parameter: client_options
+  def self.scrape_period(url:, period:, webguest: "P1.WEBGUEST",
+                         client_options: {}
   )
-    agent = ScraperUtils::MechanizeUtils.mechanize_agent(use_proxy: use_proxy, timeout: timeout)
-    agent.verify_mode = OpenSSL::SSL::VERIFY_NONE if disable_ssl_certificate_check
-    
-    # ... rest of code ...
-
-    # Update yield to return unprocessable as well as record
+    agent = ScraperUtils::MechanizeUtils.mechanize_agent(**client_options)
 
+    # ... rest of code ...
   end
+
   # ... rest of code ...
 end
 ```
 
 ### Debugging Techniques
 
-The following code will print dbugging info if you set:
+The following code will cause debugging info to be output:
 
 ```bash
 export DEBUG=1
@@ -235,8 +275,8 @@ require 'scraper_utils'
 
 # Debug an HTTP request
 ScraperUtils::DebugUtils.debug_request(
-  "GET", 
-  "https://example.com/planning-apps", 
+  "GET",
+  "https://example.com/planning-apps",
   parameters: { year: 2023 },
   headers: { "Accept" => "application/json" }
 )
@@ -248,7 +288,56 @@ ScraperUtils::DebugUtils.debug_page(page, "Checking search results page")
 ScraperUtils::DebugUtils.debug_selector(page, '.results-table', "Looking for development applications")
 ```
 
-## Development
+Interleaving Requests
+---------------------
+
+The `ScraperUtils::FiberScheduler` provides a lightweight utility that:
+
+* works on the other authorities whilst in the delay period for an authorities next request
+* thus optimizing the total scraper run time
+* allows you to increase the random delay for authorities without undue effect on total run time
+* For the curious, it uses [ruby fibers](https://ruby-doc.org/core-2.5.8/Fiber.html) rather than threads as that is
+  simpler to get right and debug!
+
+To enable change the scrape method in the example above to;
+
+```ruby
+
+def scrape(authorities, attempt)
+  ScraperUtils::FiberScheduler.reset!
+  exceptions = {}
+  authorities.each do |authority_label|
+    ScraperUtils::FiberScheduler.register_operation(authority_label) do
+      ScraperUtils::FiberScheduler.log "Collecting feed data for #{authority_label}, attempt: #{attempt}..."
+      begin
+        ScraperUtils::DataQualityMonitor.start_authority(authority_label)
+        YourScraper.scrape(authority_label) do |record|
+          begin
+            record["authority_label"] = authority_label.to_s
+            ScraperUtils::DbUtils.save_record(record)
+          rescue ScraperUtils::UnprocessableRecord => e
+            ScraperUtils::DataQualityMonitor.log_unprocessable_record(e, record)
+            exceptions[authority_label] = e
+          end
+        end
+      rescue StandardError => e
+        warn "#{authority_label}: ERROR: #{e}"
+        warn e.backtrace
+        exceptions[authority_label] = e
+      end
+    end # end of register_operation block
+  end
+  ScraperUtils::FiberScheduler.run_all
+  exceptions
+end
+```
+
+And use `ScraperUtils::FiberScheduler.log` instead of `puts` when logging within the authority processing code.
+This will prefix the output lines with the authority name, which is needed since the system will interleave the work and
+thus the output.
+
+Development
+-----------
 
 After checking out the repo, run `bin/setup` to install dependencies.
 Then, run `rake test` to run the tests.
@@ -259,13 +348,19 @@ To install this gem onto your local machine, run `bundle exec rake install`.
 
 To release a new version, update the version number in `version.rb`, and
 then run `bundle exec rake release`,
-which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+which will create a git tag for the version, push git commits and tags, and push the `.gem` file
+to [rubygems.org](https://rubygems.org).
+
+NOTE: You need to use ruby 3.2.2 instead of 2.5.8 to release to OTP protected accounts.
 
-## Contributing
+Contributing
+------------
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/ianheggie-oaf/scraper_utils
 
-## License
+CHANGELOG.md is maintained by the author aiming to follow https://github.com/vweevers/common-changelog
 
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+License
+-------
 
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/SPECS.md

@@ -0,0 +1,25 @@
+SPECS
+=====
+
+These project specific Specifications go into further details than the
+installation and usage notes in `README.md`.
+
+ASK for clarification of any apparent conflicts with IMPLEMENTATION, GUIDELINES or project instructions.
+
+## Core Design Principles
+
+### Error Handling
+- Record-level errors abort only that record's processing
+- Allow up to 5 + 10% unprocessable records before failing
+- External service reliability (e.g., robots.txt) should not block core functionality
+
+### Rate Limiting
+- Honor site-specific rate limits when clearly specified
+- Apply adaptive delays based on response times
+- Use randomized delays to avoid looking like a bot
+- Support proxy configuration for geolocation needs
+
+### Testing
+- Ensure components are independently testable
+- Avoid timing-based tests in favor of logic validation
+- Keep test scenarios focused and under 20 lines

data/bin/console

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require "bundler/setup"
 require "scraper_utils"

data/bin/setup

@@ -1,4 +1,5 @@
-#!/usr/bin/env bash
+#!/bin/bash
+
 set -euo pipefail
 IFS=$'\n\t'
 set -vx