bot_challenge_page 0.4.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1f123fae1f11aef11705e08404a9275bf61e9657cac5bc2833145494a26cbb10
-  data.tar.gz: 9ea4ba0896da4e3dc7ca4307fff1c18cc2f0b33a7c792d11f3ed926559f183a8
+  metadata.gz: cb4f4b1a29eae8a320d95d2279e21ab44ef8168be416136790c96d2150de1ac3
+  data.tar.gz: fa8fe854808413b95c4e2c4a5c4ecf4ed7b3826f77b487d46aee895f9a2d6735
 SHA512:
-  metadata.gz: 1e5ef5eaedf5c9705270da5b55ac3c0f8cf225b22ba09af3bb0782afcbc12ade3bc02b8c642ffaae14ca0c401a5858344c404c144d2094cf9f372f23ae5189c2
-  data.tar.gz: e65401e9d3a5b8379fad737842138312d5f3d3f003e59123246cbb0c0af9de932058736f07ba4b3f161eb39bdbe228e941635f63e861a7dcb0e465a1d7e6c272
+  metadata.gz: '082b26cdef3709185fb7616d009798c26d244485ccb03e24593512f1af6481450413351356f044e380e953f65e28a6ac386ca490ba778fbea78c71109f2dc98d'
+  data.tar.gz: 805a64dfc7de28b6f27fdf0c4b64b18cba29e98ae59376bce6565e133b993246b3783312f65ad69d9c65df8ea31a4193f0dbe2cd5cb4006ca81264caa60464c7

data/README.md

@@ -2,16 +2,15 @@
 
 [![CI](https://github.com/samvera-labs/bot_challenge_page/actions/workflows/ci.yml/badge.svg)](https://github.com/samvera-labs/bot_challenge_page/actions/workflows/ci.yml) [![Gem Version](https://badge.fury.io/rb/bot_challenge_page.png)](http://badge.fury.io/rb/bot_challenge_page)
 
-BotChallengePage lets you protect certain routes in your Rails app with [CloudFlare Turnstile](https://www.cloudflare.com/application-services/products/turnstile/) "CAPTHCA alternate" bot detector. Rather than the typical form submission use case for Turnstile, the user will be redirected to an interstitial challenge page, and automatically redirected back immediately on success.
+BotChallengePage lets you protect certain **GET** routes in your Rails app with [CloudFlare Turnstile](https://www.cloudflare.com/application-services/products/turnstile/) "CAPTHCA alternate" bot detector. Rather than the typical form submission use case for Turnstile, the user will be redirected to an interstitial challenge page, and automatically redirected back immediately on success.
 
-The motivating use case is fairly dumb (probably AI-related) crawlers, rather than targetted attacks, although we have tried to pay attention to security.  Many of our use cases were crawlers getting caught following every combination of voluminous facet values in search results in a near "infinite space", and causing us resource usage issues.
+The motivating use case is fairly dumb (probably AI-related) crawlers crawling search results pages, rather than targetted attacks, although we have tried to pay attention to security.  Many of our use cases were crawlers getting caught following every combination of voluminous facet values in search results in a near "infinite space", and causing us resource usage issues.
 
 ![challenge page screenshot](docs/challenge-page-example.png)
 
-* You can optionally configure a rate limit that is allowed BEFORE the challenge is triggered
-  * Uses rack-attack to track rate, requires `Rails.cache` or `Rack::Attack.cache.store` to be set to a persistent shared high-performance cache, probably redis or memcached.
+* Support both immediate bot challenge, or optionally a rate limit that will trigger a bot challenge.
 
-* Once a challenge is passed, the pass is stored in a cookie, and a challenge won't be redisplayed for a configurable amount of time, so long as cookie is present
+* Once a challenge is passed, the pass is stored in a cookie, and a challenge won't be redisplayed for a configurable amount of time, so long as cookie is present, and client matches a configurable user-agent/IP address fingerprint.
 
 * **Note:** User-agent does always need both cookies and javascript enabled to be able to pass challenge and get through!
 
@@ -23,30 +22,64 @@ The motivating use case is fairly dumb (probably AI-related) crawlers, rather th
 * `bundle add bot_challenge_page`, `bundle install`
 
 * Run the installer
-  * if you want to use rack-attack for some permissive pre-challenge rate, `rails g bot_challenge_page:install`
+  * `rails g bot_challenge_page:install`
+  * This will add a line to your ApplicationController to include a mixin to provide a `bot_challenge` configuration method in your controllers
+  * And a template configuration page at `./config/initializers/bot_challenge_page.rb`
 
-  * If you do not want to use rack-attack and want challenge on FIRST request, `rails g bot_challenge_page:install --no-rack-attack`
 
-  * By default challenge pages are "inline" at protected URL. To redirect to a separate challenge page URL instead, `--redirect-for-challenge`
 
-* If you are **not using rack-attack**, you need to add a before_action to the controller(s)
-  you'd like to protect, eg:
+* Configure in the generated `./config/initializers/bot_challenge_page.rb`
+  * At a minimum you need to configure your Cloudflare Turnstile keys
 
-        before_action only: :index do |controller|
-          BotChallengePage::BotChallengePageController.bot_challenge_enforce_filter(controller, immediate: true)
-        end
+  * Some other configuration options are offered -- more advanced/specialized ones are available that are not mentioned in generated config file, see [Config class](./app/models/bot_challenge_page/config.rb)
 
+## Protect some paths
+
+You can add `bot_challenge` to a controller to protect all actions in that controller with a bot challenge.
+
+You can also use all the Rails `before_action` params to apply to only some actions or requests in that controller: `only` and `except` to specify actions; and `if` and `unless` to specify procs to filter individual requests.
 
-* Configure in the generated `./config/initializers/bot_challenge_page.rb`
-  * At a minimum you need to configure your Cloudflare Turnstile keys, and some paths to protect!
     * Note that we can only protect GET paths, and also think about making sure you DON'T protect
       any path your front-end needs JS `fetch` access to, as this would block it (at least
       without custom front-end code we haven't really explored)
 
-    * If you are tempted to just protect `/` that may work, but worth thinking about any hearbeat paths, front-end requestable paths, or other machine-access-desired paths.
+    * If you are tempted to just protect `/` that may work, but you may need to exclude hearbeat paths, front-end (AJAX) requestable paths, API endpoints, uptime checker requests, or other machine-access-desired paths. These may be good candidates for an `unless` parameter, or the `skip_when` configuration.
 
-  * Some other configuration options are offered -- more advanced/specialized ones are available that are not mentioned in generated config file, see [Config class](./app/models/bot_challenge_page/config.rb)
+    * The author is a librarian who believes maintaining machine access in general is a public good, and tries to limit access with a bot challenge to the minimum paths necessary for app sustainability.
+
+    * The default configuration only allows re-use of a 'pass' cookie from requests with same IP address subnet and user-agent-related headers. This can be customized.
+
+```ruby
+class WidgetController < ApplicationController
+  bot_challenge only: :index, unless: -> { headers['x-secret-code'] == "i_am_uptime-checker" }
+end
+```
+
+### Protect some paths with a rate limit
+
+If you want to display a bot challenge only after some rate is reached, you will need some [Rails cache store configured](https://guides.rubyonrails.org/caching_with_rails.html#configuration) to keep track of rate.  You can configure `Rails.config.cache`, or for bot_challenge_page specifically in it's config.
+
+* Redis or Memcached are typical, but the `memory_store` cache can work if you don't mind your rate limits being only approximate -- they will reset on every web server process restart, and if you have more than one web server process they will each have their own rate limit.
+
+You use the `after` and `within` argument to `bot_challenge` to include a rate limit. `only`, `except`, `if`, and `unless` are still supported.
+
+```ruby
+class WidgetController < ApplicationController
+  bot_challenge after: 2, within: 3.hours, only: :index, if: -> { request_has_facet_limits? }
+end
+```
+
+#### rate limit counters
+
+By default, all `bot_challenge` directives share a rate limit counter. So if two differnet controllers have a `bot_challenge`, requests to either one add to the same counter for rate limit checks.
+
+Which also means if you have more than one `bot_challenge` that can apply to the _same request_, it might get double-counted (or more-counted).  (Also too many rate-limited bot_challenges applying to the same request could have performance implications).
 
+To avoid this problem or achieve desired behavior, you can pass a `counter` string into `bot_challenge` to declare separate counteres and decide which `bot_challenge` should or should not share counters.
+
+The `counter` arg has overlapping use but distinct effect from passing in a `by` argument or setting `config.default_limit_by`, which lets you determine how user-agents are identified to share a counter bucket, which by default buckets clients by IP subnet, not just individual IP.
+
+If a given request does not apply to `bot_challenge` because of `only`, `except`, `if`, `unless` or `config.skip_when` -- it **does not count toward rate limit either**.
 
 ## Customize challenge page display
 
@@ -101,42 +134,66 @@ end
 Many of us in my professional community use [blacklight](https://github.com/projectblacklight/blacklight).  Here's a possible sample blacklight config to:
 
 * Protect default catalog controller, including search results and any other actions
-* But give the user 3 free searches in a 36 hour period before challenged
-* For the #facet action used for "facet… more" links --  exempt from protection if the request is being made by a browser JS `fetch`, we just let those through. (Which means a determined attacker could do that on purpose, not defense against on purpose DDoS)
+* ONLY if the search includes a query string or facet limit -- allow unfiltered search, including pagination, without bot challenge.
+* Even for queried or limite results, give an IP subnet 1 free searches in a 36 hour period before challenged
+* For the action used for "facet… more" links and `blacklight_range_limit` that need to be XHR/JS-fetchable --  exempt from protection if the request is being made by a browser JS `fetch`, we just let those through. (Which means a determined attacker could do that on purpose, not defense against on purpose DDoS)
+* Let's an uptime checker in based on secret code in headers
+
+
 
 ```ruby
+# ./config/initializers/bot_challenge_page.rb
 Rails.application.config.to_prepare do
   BotChallengePage::BotChallengePageController.bot_challenge_config.enabled = true
 
+  # Need to set store to a Rails cache store other than null store, if you want to track
+  # rate limits.
+  BotChallengePage::BotChallengePageController.bot_challenge_config.store = :redis_store
+
   # Get from CloudFlare Turnstile: https://www.cloudflare.com/application-services/products/turnstile/
   BotChallengePage::BotChallengePageController.bot_challenge_config.cf_turnstile_sitekey = "MUST GET"
   BotChallengePage::BotChallengePageController.bot_challenge_config.cf_turnstile_secret_key = "MUST GET"
 
-  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limited_locations = [
-    "/catalog"
-  ]
-
-  # allow rate_limit_count requests in rate_limit_period, before issuing challenge
-  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limit_period = 36.hour
-  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limit_count = 3
-
-  BotChallengePage::BotChallengePageController.allow_exempt = ->(controller, config) {
-    # Excempt any Catalog #facet or #range_limit action that looks like an ajax/fetch request, the # challenge isn't going to work there, we just exempt it.
-    #
-    # sec-fetch-dest is set to 'empty' by browser on fetch requests, to limit us further;
-    # sure an attacker could fake it, we don't mind if someone determined can avoid
-    # bot challenge on this one action
-    ( controller.params[:action].in?(["facet", "range_limit"]) &&
-      controller.request.headers["sec-fetch-dest"] == "empty" &&
-      controller.kind_of?(CatalogController)
+  BotChallengePage::BotChallengePageController.bot_challenge_config.skip_when = ->(config) {
+    # Exempt honeybadger token to allow HB uptime checker in
+    # https://docs.honeybadger.io/guides/security/
+    (
+      ENV['HONEYBADGER_TOKEN'].present? &&
+      controller.request.headers['Honeybadger-Token'] == ENV['HONEYBADGER_TOKEN']
     )
   }
 
-  BotChallengePage::BotChallengePageController.rack_attack_init
 end
-
 ```
 
+```ruby
+# ./app/controllers/catalog_controller.rb
+class CatalogController < ApplicationController
+  # from default blacklight first...
+  include Blacklight::Catalog
+  include BlacklightRangeLimit::ControllerOverride
+
+   # This should apply to all CatalogController sub-classes too, which include CollectionShowController and
+  # FeaturedTopicController. They all share a counter though.
+  #
+  # We let bots through if they have NO query params, we want let collection/focus sploash
+  # pages be indexed -- this will actually let bot paginate through entire results with
+  # no query/facets, which we seem to be able to tolerate.
+  #
+  bot_challenge after: 1, within: 12.hours,
+    if: -> {
+      has_search_parameters?
+    },
+    except: ["facet", "range_limit"]
+
+  # facet and range_limit both get challenged immediately, unless they are JS fetch,
+  # in which case they are let in freely.
+  bot_challenge only: ["facet", "range_limit"], unless: -> {
+    request.headers["sec-fetch-dest"] == "empty"
+  }
+
+end
+
 ## Development and automated testing
 
 All logic and config hangs off a controller, with the idea that you could sub-class the controller to override any functionality -- or even have multiple sub-classes in your app with different configuration or customized config. But this hasn't really been tested/fleshed out yet.
@@ -153,12 +210,11 @@ If you make any changes to `Gemfile` you may need to run `bundle exec appraisal
 
 ## Possible future features?
 
-* allow regex in default location_matcher? Easy to do if you want it, just say so.
-
 * We could support swap-in Turnstile-alternatives, like [hCAPTHCA](https://www.hcaptcha.com/), [Google reCAPTCHA v3](https://developers.google.com/recaptcha/docs/v3), or even open source proof of work implementations like [ALTCHA](https://altcha.org/docs/get-started/), [pow-bot-deterrent](https://github.com/sequentialread/pow-bot-deterrent), or [Friendly Captcha](https://github.com/FriendlyCaptcha/friendly-captcha-sdk).  But the (free) cost/benefit of Turnstile are pretty good, so I don't myself have a lot of motivation to add this complexity.
 
 * Something to make it easier to switch the challenge on only based on signals that server/app is under some defined heavy load?
 
+* Use the in-development [bot auth](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/web-bot-auth/) standard, to support allow-listing of specified auth\'d bots.
 
 ## License
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/app/controllers/bot_challenge_page/bot_challenge_page_controller.rb

@@ -11,20 +11,17 @@ require 'http'
 #
 module BotChallengePage
   class BotChallengePageController < ::ApplicationController
-    include BotChallengePage::RackAttackInit
-    include BotChallengePage::EnforceFilter
+    include BotChallengePage::GuardAction
 
-    # Config for bot detection is held in class object here -- idea is
-    # to support different controllers with different config protecting
-    # different paths in your app if you like, is why config is with controller
-    class_attribute :bot_challenge_config, default: ::BotChallengePage::Config.new
+    # We access all config at the controller level, intending to support
+    # a design of different controllers in the same app with different config, protecting
+    # different parts of your app.
+    #
+    # But most people won't use that, just default to a global config for simplicity.
+    class_attribute :bot_challenge_config, default: ::BotChallengePage.config
 
     SESSION_DATETIME_KEY = "t"
-    SESSION_IP_KEY = "i"
-
-    # for allowing unsubscribe for testing
-    class_attribute :_track_notification_subscription, instance_accessor: false
-
+    SESSION_FINGERPRINT_KEY = "f"
 
     # only used if config.redirect_for_challenge is true
     def challenge
@@ -58,7 +55,7 @@ module BotChallengePage
         Rails.logger.info("#{self.class.name}: Cloudflare Turnstile validation passed api (#{request.remote_ip}, #{request.user_agent}): #{params["dest"]}")
         session[self.bot_challenge_config.session_passed_key] = {
           SESSION_DATETIME_KEY => Time.now.utc.iso8601,
-          SESSION_IP_KEY   => request.remote_ip
+          SESSION_FINGERPRINT_KEY   => self.bot_challenge_config.session_valid_fingerprint.call(request)
         }
       else
         Rails.logger.warn("#{self.class.name}: Cloudflare Turnstile validation failed (#{request.remote_ip}, #{request.user_agent}): #{result}: #{params["dest"]}")

data/app/controllers/concerns/bot_challenge_page/controller.rb

@@ -0,0 +1,95 @@
+module BotChallengePage
+  # To be included in app controllers to make `bot_challenge` class method macro available.
+  module Controller
+    extend ActiveSupport::Concern
+
+    class_methods do
+      def bot_challenge(challenge_controller: BotChallengePage::BotChallengePageController,
+                        after:nil,
+                        within:nil,
+                        by: ->{
+                          instance_exec(challenge_controller.bot_challenge_config, &challenge_controller.bot_challenge_config.default_limit_by)
+                        },
+                        store: nil,
+                        counter: nil,
+                        **before_action_options)
+
+
+
+        unless_arg = before_action_options.delete(:unless)
+        generated_unless = -> {
+          (unless_arg && instance_exec(&unless_arg)) ||
+          instance_exec(challenge_controller.bot_challenge_config, &challenge_controller.bot_challenge_config.skip_when)
+        }
+
+        if after
+          unless within
+            raise ArgumentError.new("either both or neither of `after` and `within` must be speciied")
+          end
+
+          self._bot_challenge_rate_limit(to: after, within: within, by: by,  store: store,
+            context: ["bot_challenge", counter].compact.join('.'),
+            with: ->{
+              challenge_controller.bot_challenge_guard_action(self)
+            },
+            unless: generated_unless,
+            **before_action_options)
+        else
+          before_action(unless: generated_unless, **before_action_options) do
+            ActiveSupport::Notifications.instrument("before_action.bot_challenge_page", request: request) do
+              challenge_controller.bot_challenge_guard_action(self)
+            end
+          end
+        end
+      end
+
+
+      # A copy-paste-customize of Rails rate_limit at
+      # https://github.com/rails/rails/blob/9a64857d7002554b0af94158de386def5bfef9d3/actionpack/lib/action_controller/metal/rate_limiting.rb#L55
+      #
+      # For two purposes:
+      #
+      # 1. Apply 'context' argument from https://github.com/rails/rails/pull/55299 (not merged when I write this)
+      #
+      # 2. Make 'store' defaults calculated _at execution time_ rather than definition time, which is
+      #    convenient for being able to mock config in applicaiton tests.
+      #
+      def _bot_challenge_rate_limit(to:, within:, by: -> { request.remote_ip }, with: -> { head :too_many_requests }, store: nil, name: nil, context: nil,
+            challenge_controller: BotChallengePage::BotChallengePageController,  # to get config for store default
+            **options)
+        before_action -> {
+          _bot_challenge_rate_limiting(to: to,
+                                        within: within,
+                                        by: by,
+                                        with: with,
+                                        store: store || challenge_controller.bot_challenge_config.store || cache_store,
+                                        name: name,
+                                        context: context)
+        }, **options
+      end
+    end
+
+    private
+
+    # See above at _bot_challenge_rate_limit
+    #
+    def _bot_challenge_rate_limiting(to:, within:, by:, with:, store:, name:, context:)
+      by = instance_exec(&by)
+      cache_key = ["rate-limit", context || controller_path, name, by].compact.join(":")
+      count = store.increment(cache_key, 1, expires_in: within)
+      if count && count > to
+        ActiveSupport::Notifications.instrument("rate_limit.action_controller",
+            request: request,
+            count: count,
+            to: to,
+            within: within,
+            by: by,
+            name: name,
+            context: context,
+            cache_key: cache_key) do
+          instance_exec(&with)
+        end
+      end
+    end
+  end
+end

data/app/controllers/concerns/bot_challenge_page/{enforce_filter.rb → guard_action.rb}

@@ -3,21 +3,18 @@ module BotChallengePage
   # Extracted to concern in separate file mostly for readability, not expected to be used
   # anywehre but BotChallengePageController -- we hang all logic off controller to allow multiple
   # controllers in an app, and over-ride in sub-classes.
-  module EnforceFilter
+  module GuardAction
     extend ActiveSupport::Concern
 
     class_methods do
-      # Usually in your ApplicationController, unless using `immediate`.
+      # All the logic for enforcing bot challenge protection, usually in a before_filter
+      # of some kind, direct or rate_limit.
       #
-      #     before_action { |controller| BotChallengePage::BotChallengePageController.bot_challenge_enforce_filter(controller) }
-      #
-      # @param immediate [Boolean] always force bot protection, ignore any allowed pre-challenge rate limit
-      def bot_challenge_enforce_filter(controller, immediate: false)
+      # Render challenge page when necessary, otherwise do nothing allowing ordinary rails render.
+      def bot_challenge_guard_action(controller)
         if self.bot_challenge_config.enabled &&
-            (controller.request.env[self.bot_challenge_config.env_challenge_trigger_key] || immediate) &&
             ! self._bot_detect_passed_good?(controller.request) &&
-            ! controller.kind_of?(self) && # don't ever guard ourself, that'd be a mess!
-            ! self.bot_challenge_config.allow_exempt.call(controller, self.bot_challenge_config)
+            ! controller.kind_of?(self) # don't ever guard ourself, that'd be a mess!
 
           # we can only do GET requests right now
           if !controller.request.get?
@@ -49,10 +46,12 @@ module BotChallengePage
 
         return false unless session_data && session_data.kind_of?(Hash)
 
-        datetime = session_data[BotChallengePageController::SESSION_DATETIME_KEY]
-        ip   = session_data[BotChallengePageController::SESSION_IP_KEY]
+        datetime = session_data[self::SESSION_DATETIME_KEY]
+
+        fingerprint   = session_data[self::SESSION_FINGERPRINT_KEY]
 
-        (ip == request.remote_ip) && (Time.now - Time.iso8601(datetime) < self.bot_challenge_config.session_passed_good_for )
+        (Time.now - Time.iso8601(datetime) < self.bot_challenge_config.session_passed_good_for ) &&
+        fingerprint == self.bot_challenge_config.session_valid_fingerprint.call(request)
       end
     end
   end

data/app/views/bot_challenge_page/pow1/_pow1_placeholder.html.erb

@@ -0,0 +1,9 @@
+<% calculator = BotChallengePage::SimplePow1.new(client_id: request.remote_ip) %>
+
+<div
+  data-bot-challenge="pow1"
+  data-pow1-challenge="<%= calculator.challenge %>"
+  data-pow1-difficulty="<%= calculator.difficulty %>"
+>
+Testing your stuff
+</div>

data/app/views/bot_challenge_page/pow1/_pow1_script_tag.html.erb

@@ -0,0 +1,121 @@
+<script type="text/javascript" async defer>
+
+
+async function execute() {
+  try {
+    const placeholder = document.querySelector("*[data-bot-challenge='pow1']");
+
+    const challenge = placeholder.dataset.pow1Challenge;
+    const difficulty = placeholder.dataset.pow1Difficulty;
+
+    const solution_base64 = await process(challenge, difficulty);
+
+    const csrfToken = document.querySelector("[name='csrf-token']");
+    const response = await fetch('<%= bot_detect_challenge_path %>', {
+      method: 'POST',
+      headers: {
+        "X-CSRF-Token": csrfToken?.content,
+        "Content-Type": "application/json"
+      },
+      body: JSON.stringify({
+        pow1_solution_base64: solution_base64,
+        difficulty: difficulty
+      }),
+    });
+
+    processFetchResponse(response);
+  }
+  catch(error) {
+    console.error("Error processing turnstile challenge backend action: " + error);
+    _displayChallengeError();
+  }
+}
+
+// Returns a solution as base64 encoded uint8Array
+async function process(hexChallenge, difficulty = 18) {
+  const uint8Challenge = uint8Array_fromHex(hexChallenge);
+
+  let solution;
+  let bitPrefixArr;
+  let solutionNum = 0;
+  let digestUint8array;
+  do {
+      solutionNum = solutionNum + 1;
+      solution = intToUInt8Array(solutionNum);
+      digestUint8array = await sha256(mergeUInt8Arrays(solution, uint8Challenge));
+
+      bitPrefixArr = getFirstXBits(digestUint8array, difficulty);
+  } while( !(bitPrefixArr.length == difficulty && bitPrefixArr.every(element => element === 0)) )
+
+
+
+console.log("solution hex: " + uInt8Array_toHex(solution));
+console.log("challenge hex: " + hexChallenge);
+console.log("solution + challenge (hex): " + uInt8Array_toHex(mergeUInt8Arrays(solution, uint8Challenge)));
+console.log("combined sha hex: " + uInt8Array_toHex(digestUint8array));
+console.log("bit prefix arr: " + bitPrefixArr);
+
+  return uint8Array_toBase64(solution);
+}
+
+// Mostly for debugging
+function uInt8Array_toHex(buffer) {
+  return Array.from(buffer)
+    .map(x => x.toString(16).padStart(2, '0'))
+    .join('');
+}
+
+// Uint8Array => Uint8Array
+async function sha256(uint8_array) {
+  return new Uint8Array(await crypto.subtle.digest('SHA-256', uint8_array));
+}
+
+function intToUInt8Array(i) {
+  // Got me, but https://stackoverflow.com/a/78592211/307106
+  return new Uint8Array(new BigUint64Array([BigInt(i)]).buffer);
+}
+
+function mergeUInt8Arrays(a1, a2) {
+  // sum of individual array lengths
+  var mergedArray = new Uint8Array(a1.length + a2.length);
+  mergedArray.set(a1);
+  mergedArray.set(a2, a1.length);
+  return mergedArray;
+}
+
+function uint8Array_fromHex(hexString) {
+  // https://stackoverflow.com/a/50868276/307106
+  return Uint8Array.from(hexString.match(/.{1,2}/g).map((byte) => parseInt(byte, 16)));
+}
+
+// https://stackoverflow.com/a/66046176/307106
+async function uint8Array_toBase64(buffer) {
+  // use a FileReader to generate a base64 data URI:
+  const base64url = await new Promise(r => {
+    const reader = new FileReader()
+    reader.onload = () => r(reader.result)
+    reader.readAsDataURL(new Blob([buffer]))
+  });
+  // remove the `data:...;base64,` part from the start
+  return base64url.slice(base64url.indexOf(',') + 1);
+}
+
+//https://stackoverflow.com/a/78592211/307106
+function getFirstXBits(uint8Array, x) {
+  const numBytes = Math.ceil(x / 8);
+  const view = uint8Array.subarray(0, numBytes);
+  const bits = [];
+  let bitIndex = 0;
+
+  for (const byte of view) {
+    for (let i = 7; i >= 0 && bitIndex < x; i--) {
+      bits.push((byte >> i) & 1);
+      bitIndex++;
+    }
+  }
+  return bits;
+}
+
+<%= render partial: "bot_challenge_page/common_func", formats: [:js] %>
+
+</script>

data/{app/models → lib}/bot_challenge_page/config.rb

@@ -13,17 +13,7 @@ module BotChallengePage
 
     def initialize(**values)
       self.class.attr_defaults.merge(values).each_pair do |key, value|
-        # super hacky way to execute any procs in the context of this config,
-        # so they can access other config values easily.
-        if value.kind_of?(Proc)
-          newval = lambda do |*args|
-            self.instance_exec(*args, &value)
-          end
-        else
-          newval = value
-        end
-
-        send("#{key}=", newval)
+        send("#{key}=", value)
       end
     end
 
@@ -31,32 +21,22 @@ module BotChallengePage
     # with a 403 status (false)
     attribute :redirect_for_challenge, default: false
 
-    attribute :enabled, default: false # Must set to true to turn on at all
+    attribute :enabled, default: true
+
+    # ActiveSupport::Cache::Store to use for rate info, if nil will use Controller #cache_store
+    attribute :store
 
     attribute :cf_turnstile_sitekey, default: "1x00000000000000000000AA" # a testing key that always passes
     attribute :cf_turnstile_secret_key, default: "1x0000000000000000000000000000000AA" # a testing key always passes
     # Turnstile testing keys: https://developers.cloudflare.com/turnstile/troubleshooting/testing/
 
-    # up to rate_limit_count requests in rate_limit_period before challenged
-    attribute :rate_limit_period,  default: 12.hour
-    attribute :rate_limit_count,  default: 10
-
     # how long is a challenge pass good for before re-challenge?
     attribute :session_passed_good_for,  default: 24.hours
 
-    # An array, can be:
-    #   * a string, path prefix
-    #   * a hash of rails route-decoded params, like `{ controller: "something" }`,
-    #     or `{ controller: "something", action: "index" }
-    #     The hash is more expensive to check and uses some not-technically-public
-    #     Rails api, but it's just so convenient.
-    #
-    # Used by default :location_matcher, if set custom may not be used
-    attribute :rate_limited_locations, default: []
 
-    # Executed at the _controller_ filter level, to last minute exempt certain
-    # actions from protection.
-    attribute :allow_exempt, default: ->(controller, config) { false }
+    # Executed inside a controller instance, to omit a request from bot challenge.
+    # Adds on to :unless arg.
+    attribute :skip_when, default: ->(config) { false }
 
     # replace with say `->() { render layout: 'something' }`, or `render "somedir/some_template"`
     attribute :challenge_renderer, default: ->() {
@@ -69,48 +49,47 @@ module BotChallengePage
     # rate limit per subnet, follow lehigh's lead with
     # subnet: /16 for IPv4 (x.y.*.*), and /64 for IPv6 (about the same size subnet for better or worse)
     # https://git.drupalcode.org/project/turnstile_protect/-/blob/0dae9f95d48f9d8cae5a8e61e767c69f64490983/src/EventSubscriber/Challenge.php#L140-151
-    attribute :rate_limit_discriminator, default: (lambda do |req, config|
-      if req.ip.index(":") # ipv6
-        IPAddr.new("#{req.ip}/64").to_string
+    attribute :default_limit_by, default: (lambda do |config|
+      if request.remote_ip.index(":") # ipv6
+        IPAddr.new("#{request.remote_ip}/64").to_string
       else
-        IPAddr.new("#{req.ip}/16").to_string
+        IPAddr.new("#{request.remote_ip}/16").to_string
       end
     rescue IPAddr::InvalidAddressError
-      req.ip
+      req.remote_ip
     end)
 
-    attribute :location_matcher, default: ->(rack_req, config) {
-      parsed_route = nil
-      config.rate_limited_locations.any? do |val|
-        case val
-        when Hash
-          begin
-            # #recognize_path may e not techinically public API, and may be expensive, but
-            # no other way to do this, and it's mentioned in rack-attack:
-            # https://github.com/rack/rack-attack/blob/86650c4f7ea1af24fe4a89d3040e1309ee8a88bc/docs/advanced_configuration.md#match-actions-in-rails
-            # We do it lazily only if needed so if you don't want that don't use it.
-            parsed_route ||= rack_req.env["action_dispatch.routes"].recognize_path(rack_req.url, method: rack_req.request_method)
-            parsed_route && parsed_route >= val
-          rescue ActionController::RoutingError
-            false
-          end
-        when String
-          # string complete path at beginning, must end in ?, or end of string
-          /\A#{Regexp.escape val}(\/|\?|\Z)/ =~ rack_req.path
-        end
+    # fingerprint is taken when "pass" is stored in session. client
+    # fingerprint needs to be the same to use pass, or else it's rejected.
+    #
+    # Algorithm parts based on advice from Xe laso @ Anubis, with variations.
+    #
+    # Allow exact IP to change -- various IPv6 and NAT can make it -- but within limited
+    # subnet.  But also force some other headers to match, which they should if it's the same
+    # user-agent, which it should be if it's re-using a cookie.
+    attribute :session_valid_fingerprint, default: ->(request) {
+      ip_subnet_base = if request.remote_ip.index(":") #ipv6
+        IPAddr.new("#{request.remote_ip}/64").to_string
+      else
+        IPAddr.new("#{request.remote_ip}/24").to_string
       end
+
+      [
+        request.user_agent,
+        request.headers['sec-ch-ua-platform'],
+        request.headers['accept-encoding'],
+        ip_subnet_base
+      ].join(":")
     }
+
+
     attribute :cf_turnstile_js_url, default: "https://challenges.cloudflare.com/turnstile/v0/api.js"
     attribute :cf_turnstile_validation_url, default:  "https://challenges.cloudflare.com/turnstile/v0/siteverify"
     attribute :cf_timeout, default: 3 # max timeout seconds waiting on Cloudfront Turnstile api
 
-
     # key stored in Rails session object with channge passed confirmed
     attribute :session_passed_key, default: "bot_detection-passed"
 
-    # key in rack env that says challenge is required
-    attribute :env_challenge_trigger_key, default: "bot_detect.should_challenge"
-
     attribute :still_around_delay_ms, default: 1200
 
     # make sure dup dups all attributes please

data/lib/bot_challenge_page/version.rb

@@ -1,3 +1,3 @@
 module BotChallengePage
-  VERSION = "0.4.0"
+  VERSION = "0.10.0"
 end

data/lib/bot_challenge_page.rb

@@ -1,6 +1,17 @@
 require "bot_challenge_page/version"
 require "bot_challenge_page/engine"
+require "bot_challenge_page/config"
 
 module BotChallengePage
-  # Your code goes here...
+  mattr_reader :config, default: ::BotChallengePage::Config.new
+
+  # Just a convenience to allow
+  #
+  #     BotChallengePage.configure do |config|
+  #       config.foo = "bar"
+  #     end
+  #
+  def self.configure
+    yield config
+  end
 end

data/lib/generators/bot_challenge_page/install_generator.rb

@@ -13,55 +13,12 @@ module BotChallengePage
       end
     end
 
-    def add_before_filter_enforcement
-      # make the user do this themselves if they aren't using rack-attack, as it should
-      # only be on protected filters
-      return unless options[:rack_attack]
-
-      inject_into_class "app/controllers/application_controller.rb", "ApplicationController" do
-        filter_code = "BotChallengePage::BotChallengePageController.bot_challenge_enforce_filter(controller)"
-
-        <<-EOS
-  # This will only protect CONFIGURED routes, but also could be put on just certain
-  # controllers, it does not need to be in ApplicationController
-  before_action do |controller|
-    #{filter_code}
-  end
-
-        EOS
-      end
-    end
-
-    def add_rack_attack_require_if_needed
-      if options[:rack_attack]
-        # since it's an intermediate dependency, we need to require it after rails
-        # so it will load it's rails stuff
-        inject_into_file "config/application.rb", "\nrequire 'rack/attack'\n", after: /require.*rails\/[^\n]+\n/m
-
-      end
+    def add_controller_mixin
+      inject_into_class "app/controllers/application_controller.rb", "ApplicationController", "  include BotChallengePage::Controller\n"
     end
 
     def copy_initializer_file
       template "initializer.rb.erb", "config/initializers/bot_challenge_page.rb"
     end
-
-    def suggest_filter
-      unless options[:rack_attack]
-        instructions = <<~EOS
-        You must add before_action to protect controllers
-
-        Add, eg:
-
-            before_action only: :index do |controller|
-              BotChallengePage::BotChallengePageController.bot_challenge_enforce_filter(controller, immediate: true)
-            end
-
-        To desired controllers and/or ApplicationController
-        EOS
-
-        say_status("advise", instructions, :green)
-      end
-    end
-
   end
 end

data/lib/generators/bot_challenge_page/templates/initializer.rb.erb

@@ -1,56 +1,64 @@
-Rails.application.config.to_prepare do
+BotChallengePage.configure do |config|
 
-  BotChallengePage::BotChallengePageController.bot_challenge_config.enabled = true
+  # Can globally disable in configuration if desired
+  config.enabled = true
 
   # Get from CloudFlare Turnstile: https://www.cloudflare.com/application-services/products/turnstile/
   # Some testing keys are also available: https://developers.cloudflare.com/turnstile/troubleshooting/testing/
   #
   # Always pass testing sitekey: "1x00000000000000000000AA"
-  BotChallengePage::BotChallengePageController.bot_challenge_config.cf_turnstile_sitekey = "MUST GET"
+  config.cf_turnstile_sitekey = "MUST GET"
   # Always pass testing secret_key: "1x0000000000000000000000000000000AA"
-  BotChallengePage::BotChallengePageController.bot_challenge_config.cf_turnstile_secret_key = "MUST GET"
+  config.cf_turnstile_secret_key = "MUST GET"
 
-  BotChallengePage::BotChallengePageController.bot_challenge_config.redirect_for_challenge = <%= options[:redirect_for_challenge] %>
 
-  <%- if options[:rack_attack] %>
-  # What paths do you want to protect?
-  #
-  # You can use path prefixes: "/catalog" or even "/"
+  <%- if options[:redirect_for_challenge] -%>
+    config.redirect_for_challenge = <%= options[:redirect_for_challenge] %>
+  <% end %>
+
+  # For rate-limiting, we need a rails cache store that keeps state, by default
+  # will use `config.action_controller.cache_store` or Rails.cache, but if you'd
+  # like to use a separate store database, eg. :
+  # config.store = ActiveSupport::Cache::RedisCacheStore.new(url: "...")
+
+  # Filter to omit requests from bot challenge control, executed in controller instance context
   #
-  # Or hashes with controller and/or action:
+  # config.skip_when = ->(config) {
+  #   # maybe you want to globally exempt a heartbeat path
+  #   current_page?(rails_health_check_path) ||
   #
-  #   { controller: "catalog" }
-  #   { controller: "catalog", action: "index" }
+  #   # Here's a way to identify browser `fetch` API requests; note
+  #   # it can be faked by an "attacker" so you might not want to do this globally
+  #   (request.headers["sec-fetch-dest"] == "empty") ||
   #
-  # Note that we can only protect GET paths, and also think about making sure you DON'T protect
-  # any path your front-end needs JS `fetch` access to, as this would block it (at least
-  # without custom front-end code we haven't really explored)
+  #   # Maybe you want to exempt an uptime checker or other trusted bot
+  #   #based on shared secret
+  #   (headers["x-some-secret"] == "some_shared_secret")
+  # }
 
-  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limited_locations = [
-  ]
+  # Hook after a bot challenge is presented, for logging or other
+  # config.after_blocked = ->(bot_challenge_controller) {
+  # }
 
-  # allow rate_limit_count requests in rate_limit_period, before issuing challenge
-  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limit_period = 12.hour
-  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limit_count = 2
-  <% end -%>
 
   # How long will a challenge success exempt a session from further challenges?
-  # BotChallengePage::BotChallengePageController.bot_challenge_config.session_passed_good_for = 36.hours
+  # config.session_passed_good_for = 36.hours
 
-  # Exempt some requests from bot challenge protection
-  # BotChallengePage::BotChallengePageController.bot_challenge_config.allow_exempt = ->(controller) {
-  #   # controller.params
-  #   # controller.request
-  #   # controller.session
 
-  #   # Here's a way to identify browser `fetch` API requests; note
-  #   # it can be faked by an "attacker"
-  #   controller.request.headers["sec-fetch-dest"] == "empty"
-  # }
+  # Functions like to Rails rate_limit `by` parameter, as a configured default.
+  # A discriminator or identifier in which a client's requests will be bucketted
+  # by rate limit. Normally this gem buckets by IP address subnets. Switching
+  # to individual IPs would be much more generous:
+  # config.default_limit_by = ->(config) {
+  #   request.remote_ip
+  #  }
 
-  # More configuration is available
+  # When a "pass" cookie is saved, a fingerprint value is stored with it,
+  # and subsequent uses of the pass need to have a request that matches
+  # fingerprint. By default we insist on IP subnet match, and same user-agent
+  # and other headers. But can be customized.
+  # config.session_valid_fingerprint = ->(request) {
+  #    # whatever
+  # }
 
-  <%- if options[:rack_attack] %>
-  BotChallengePage::BotChallengePageController.rack_attack_init
-  <% end %>
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: bot_challenge_page
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.10.0
 platform: ruby
 authors:
 - Jonathan Rochkind
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-04-15 00:00:00.000000000 Z
+date: 2025-08-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: appraisal
@@ -30,14 +30,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '7.1'
+        version: '8.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '7.1'
+        version: '8.0'
 - !ruby/object:Gem::Dependency
   name: capybara
   requirement: !ruby/object:Gem::Requirement
@@ -114,20 +114,6 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: '8.1'
-- !ruby/object:Gem::Dependency
-  name: rack-attack
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '6.7'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '6.7'
 - !ruby/object:Gem::Dependency
   name: http
   requirement: !ruby/object:Gem::Requirement
@@ -153,16 +139,18 @@ files:
 - README.md
 - Rakefile
 - app/controllers/bot_challenge_page/bot_challenge_page_controller.rb
-- app/controllers/concerns/bot_challenge_page/enforce_filter.rb
-- app/controllers/concerns/bot_challenge_page/rack_attack_init.rb
-- app/models/bot_challenge_page/config.rb
+- app/controllers/concerns/bot_challenge_page/controller.rb
+- app/controllers/concerns/bot_challenge_page/guard_action.rb
 - app/models/bot_challenge_page/test.html
 - app/views/bot_challenge_page/_local_turnstile_script_tag.html.erb
 - app/views/bot_challenge_page/_turnstile_widget_placeholder.html.erb
 - app/views/bot_challenge_page/bot_challenge_page/challenge.html.erb
+- app/views/bot_challenge_page/pow1/_pow1_placeholder.html.erb
+- app/views/bot_challenge_page/pow1/_pow1_script_tag.html.erb
 - config/locales/bot_challenge_page.en.yml
 - config/routes.rb
 - lib/bot_challenge_page.rb
+- lib/bot_challenge_page/config.rb
 - lib/bot_challenge_page/engine.rb
 - lib/bot_challenge_page/version.rb
 - lib/generators/bot_challenge_page/install_generator.rb

data/app/controllers/concerns/bot_challenge_page/rack_attack_init.rb

@@ -1,60 +0,0 @@
-module BotChallengePage
-
-  # Extracted to concern in separate file mostly for readability, not expected to be used
-  # anywehre but BotChallengePageController -- we hang all logic off controller to allow multiple
-  # controllers in an app, and over-ride in sub-classes.
-  module RackAttackInit
-    extend ActiveSupport::Concern
-
-
-    class_methods do
-      # perhaps in an initializer, and after changing any config, run:
-      #
-      #     Rails.application.config.to_prepare do
-      #       BotChallengePage::BotChallengePageController.rack_attack_init
-      #     end
-      #
-      # Safe to call more than once if you change config and want to call again, say in testing.
-      def rack_attack_init
-        self._rack_attack_uninit # make it safe for calling multiple times
-
-        ## Turnstile bot detection throttling
-        #
-        # for paths matched by `rate_limited_locations`, after over rate_limit count requests in rate_limit_period,
-        # token will be stored in rack env instructing challenge is required.
-        #
-        # For actual challenge, need before_action in controller.
-        #
-        # You could rate limit detect on wider paths than you actually challenge on, or the same. You probably
-        # don't want to rate-limit detect on narrower list of paths than you challenge on!
-        Rack::Attack.track("bot_detect/rate_exceeded/#{self.name}",
-            limit: self.bot_challenge_config.rate_limit_count,
-            period: self.bot_challenge_config.rate_limit_period) do |req|
-          if self.bot_challenge_config.enabled && self.bot_challenge_config.location_matcher.call(req, self.bot_challenge_config)
-            self.bot_challenge_config.rate_limit_discriminator.call(req, self.bot_challenge_config)
-          end
-        end
-
-        self._track_notification_subscription = ActiveSupport::Notifications.subscribe("track.rack_attack") do |_name, _start, _finish, request_id, payload|
-          rack_request = payload[:request]
-          rack_env     = rack_request.env
-          match_name = rack_env["rack.attack.matched"]  # name of rack-attack rule
-                                                        #
-          if match_name == "bot_detect/rate_exceeded/#{self.name}"
-            match_data   = rack_env["rack.attack.match_data"]
-            match_data_formatted = match_data.slice(:count, :limit, :period).map { |k, v| "#{k}=#{v}"}.join(" ")
-            discriminator = rack_env["rack.attack.match_discriminator"] # unique key for rate limit, usually includes ip
-
-            rack_env[self.bot_challenge_config.env_challenge_trigger_key] = true
-          end
-        end
-      end
-
-      def _rack_attack_uninit
-        Rack::Attack.track("bot_detect/rate_exceeded/#{self.name}") {} # overwrite track name with empty proc
-        ActiveSupport::Notifications.unsubscribe(self._track_notification_subscription) if self._track_notification_subscription
-        self._track_notification_subscription = nil
-      end
-    end
-  end
-end