drive 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5bcf5e0361ac44acb57bedda1f651a948abd3ec6bc43cd214080556a142d4b72
-  data.tar.gz: 8c0e23b04b48848ee631f8d08a8f2c97cf15263a4ac5c4c6f58c14654de6fe1a
+  metadata.gz: c3a8ca81a5230ba8bad7ae1d5f02c1dced9130c1e4f25068a079e41bb28e082d
+  data.tar.gz: 4094ecf0fedb81c77a8764c6aedbc191e89e99400b45369a20b006a514a08663
 SHA512:
-  metadata.gz: 7a671f1e9f52a5a4422cbef14c40ab75d968be2bfe3423b9fccf3c225b6fd0227b5bfe8c301d9acd49036433b4e8efbf389a8b3068492f918aaff9b8139c15bd
-  data.tar.gz: de0351f9c2748aa969ef3fb1e2fbbf027dec38d689b733911f366f4d6e5b1e1f87ec6c7beaf365fa1cd2ab24bbd72f8e7ff42563964e37ebbbabeefd71b780cb
+  metadata.gz: b5d1ace9067f6c0ff8f7cd433742e434373ec9f4a755e31a18e37c4acc875e703ea74ce374076ebe028cc919bc347df925df25e76de8802e645111ccb76b7fe2
+  data.tar.gz: 2e0fa021cee1892ee83e6c8e335a79794c4cc1abe144e9a562516451dd63a8b7c0e202a25b61b31eb33ff7ab4aa1c791d2ac0584f017e5ac409733dee516ef2e

data/CHANGELOG.md

@@ -6,6 +6,15 @@ For more information about changelogs, check [Keep a Changelog](http://keepachan
 
 ## [Unreleased]
 
+## 0.3.0 - 2026-10-06
+
+* [FEATURE] `RateLimited` no longer limits a search engine's own crawler
+
+  `Crawler.verified?(request)` believes a user agent naming Google's, Bing's or Apple's crawler
+  only where its address resolves to their domain and that host back to the address, and
+  remembers the answer for a day. Behind a proxy, `request.remote_ip` must be the visitor's:
+  list the proxy's ranges in `config.action_dispatch.trusted_proxies`.
+
 ## 0.2.0 - 2026-10-05
 
 * [BREAKING] `drive` is a new gem: a Rails engine for apps that mirror what a source publishes

data/README.md

@@ -10,7 +10,10 @@ files, a monthly database — and tells another app exactly which rows changed.
 - **`Change`**: every row a run inserted, updated or deleted, with the row as it is now.
 - **`FeedsController`**: `GET /feed?after=ID&kind=TABLE`, the next 5,000 changes, as JSON.
 - **`AccessFile`**: stages a table of a Microsoft Access file, read with `mdb-export`.
-- **`RateLimited`**: 20 requests a minute and 200 a day per address, for public pages.
+- **`RateLimited`**: 20 requests a minute and 200 a day per address, for public pages. A search
+  engine's own crawler has no limit.
+- **`Crawler`**: whether a request comes from Google's, Bing's or Apple's crawler: its address
+  resolves to their domain and back, checked once a day per address.
 
 It needs PostgreSQL.
 

data/app/controllers/concerns/rate_limited.rb

@@ -1,9 +1,10 @@
-# The limits of a page anybody may ask for, by the address asking: 20 a minute, 200 a day.
+# The limits of a page anybody may ask for, by the address asking: 20 a minute, 200 a day. A search
+# engine's own crawler has none, so it can list every page.
 module RateLimited
   extend ActiveSupport::Concern
 
   included do
-    rate_limit to: 20, within: 1.minute, name: 'minute'
-    rate_limit to: 200, within: 1.day, name: 'day'
+    rate_limit to: 20, within: 1.minute, name: 'minute', unless: -> { Crawler.verified? request }
+    rate_limit to: 200, within: 1.day, name: 'day', unless: -> { Crawler.verified? request }
   end
 end

data/app/models/crawler.rb

@@ -0,0 +1,33 @@
+require 'resolv'
+
+# A search engine's own crawler, told apart from a client that only names one in its user agent:
+# its address resolves to the search engine's domain, and that host back to the address, which is
+# how Google and Bing say to check theirs.
+module Crawler
+  # The user agents of the crawlers checked.
+  AGENTS = /Googlebot|Google-InspectionTool|Storebot-Google|GoogleOther|bingbot|Applebot/i
+
+  # The domains their addresses resolve to.
+  DOMAINS = %w[.googlebot.com .google.com .search.msn.com .applebot.apple.com].freeze
+
+  # @param request [ActionDispatch::Request] request a page was asked with.
+  # @return [Boolean] whether it came from a crawler, checked once a day for each address.
+  def self.verified?(request)
+    ip = request.remote_ip
+    request.user_agent.to_s.match?(AGENTS) &&
+      Rails.cache.fetch(['crawler', ip], expires_in: 1.day) { resolves? ip }
+  end
+
+  # @param ip [String] address a request came from.
+  # @return [Boolean] whether it names a crawler's host, and that host names it back.
+  def self.resolves?(ip)
+    dns = Resolv::DNS.new.tap { it.timeouts = 2 }
+    host = dns.getname(ip).to_s
+    DOMAINS.any? { host.end_with? it } && dns.getaddresses(host).map(&:to_s).include?(ip)
+  rescue Resolv::ResolvError
+    false
+  ensure
+    dns&.close
+  end
+  private_class_method :resolves?
+end

data/lib/drive/version.rb

@@ -1,4 +1,4 @@
 module Drive
   # Version of the gem, read by the gemspec.
-  VERSION = '0.2.0'.freeze
+  VERSION = '0.3.0'.freeze
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: drive
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - claudiob
@@ -69,6 +69,7 @@ files:
 - app/models/change.rb
 - app/models/concerns/logged.rb
 - app/models/concerns/mirrored.rb
+- app/models/crawler.rb
 - app/models/drive/current.rb
 - app/models/run.rb
 - db/migrate/20261005100000_create_runs_and_changes.rb