drive 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c3a8ca81a5230ba8bad7ae1d5f02c1dced9130c1e4f25068a079e41bb28e082d
-  data.tar.gz: 4094ecf0fedb81c77a8764c6aedbc191e89e99400b45369a20b006a514a08663
+  metadata.gz: 99a4adcee91a1d02137f10f2ffff028ee3bcabd96896d550c71e81c443ad47b8
+  data.tar.gz: 2a5e5b329808e3e315464d70e1e8fe1357a5736632d96dde4077f3f36a57518e
 SHA512:
-  metadata.gz: b5d1ace9067f6c0ff8f7cd433742e434373ec9f4a755e31a18e37c4acc875e703ea74ce374076ebe028cc919bc347df925df25e76de8802e645111ccb76b7fe2
-  data.tar.gz: 2e0fa021cee1892ee83e6c8e335a79794c4cc1abe144e9a562516451dd63a8b7c0e202a25b61b31eb33ff7ab4aa1c791d2ac0584f017e5ac409733dee516ef2e
+  metadata.gz: 57b24c0022b7476eb3db58ab1108ee86283f5397a49704c39626f751c8158baad8d35404ceebafedaf53bbefc912d5a99e65ddbe6cd6cfea546b8417b20320d8
+  data.tar.gz: eeaa76fd2453f73da3332ded46d662d4165b458ea9db7359582ec6c05b980db96446eaccf47ed880893d1f4298fa37876c9fb573a7b9d399953d43daa98a29a4

data/CHANGELOG.md

@@ -6,6 +6,26 @@ For more information about changelogs, check [Keep a Changelog](http://keepachan
 
 ## [Unreleased]
 
+## 0.5.0 - 2026-10-07
+
+* [FEATURE] `RateLimited` no longer limits Claude's, ChatGPT's or Perplexity's search crawler
+
+  `Crawler.verified?(request)` believes a user agent naming Claude-SearchBot, OAI-SearchBot or
+  PerplexityBot, as well as Google's, Bing's or Apple's crawler, only where its address is in the
+  ranges its search engine publishes as JSON, fetched once a day. None of the three new ones
+  resolves to a domain of its own, so no crawler is checked by DNS any more.
+
+## 0.4.0 - 2026-10-06
+
+* [BREAKING] No more log of changes, and no feed
+
+  Nothing followed the feed: the apps that read a mirror read its tables directly, from the same
+  database. `Change`, `Change.prune`, `FeedsController` and the `changes` table go;
+  `bin/rails drive:install:migrations db:migrate` copies and runs the migration that drops them.
+  `Mirrored` counts the rows a refresh inserts, updates and deletes instead of logging them, and
+  `Logged` is now `Counted`, which counts each save and delete in the run at hand; a table
+  including it no longer needs `KEY` or `SOURCE`.
+
 ## 0.3.0 - 2026-10-06
 
 * [FEATURE] `RateLimited` no longer limits a search engine's own crawler

data/README.md

@@ -1,19 +1,18 @@
 # Drive
 
 A Rails engine for an app that keeps a copy of what somebody else publishes — a government's
-files, a monthly database — and tells another app exactly which rows changed.
+files, a monthly database — changing only the rows the source did, and counting them.
 
 - **`Mirrored`**: a table copying one of the source's tables field for field, as text. A refresh
   stages the source's table, then inserts, updates and deletes only the rows the source did.
-- **`Logged`**: a table written a record at a time, such as a document fetched by URL.
-- **`Run`**: one refresh, in one transaction, the only one running.
-- **`Change`**: every row a run inserted, updated or deleted, with the row as it is now.
-- **`FeedsController`**: `GET /feed?after=ID&kind=TABLE`, the next 5,000 changes, as JSON.
+- **`Counted`**: a table written a record at a time, such as a document fetched by URL.
+- **`Run`**: one refresh, in one transaction, the only one running, with how many rows it
+  inserted, updated and deleted.
 - **`AccessFile`**: stages a table of a Microsoft Access file, read with `mdb-export`.
 - **`RateLimited`**: 20 requests a minute and 200 a day per address, for public pages. A search
   engine's own crawler has no limit.
-- **`Crawler`**: whether a request comes from Google's, Bing's or Apple's crawler: its address
-  resolves to their domain and back, checked once a day per address.
+- **`Crawler`**: whether a request comes from the crawler of Google, Bing, Apple, Claude, ChatGPT
+  or Perplexity: its address is in the ranges they publish, fetched once a day.
 
 It needs PostgreSQL.
 
@@ -25,12 +24,12 @@ gem install drive
 
 ```ruby
 # Gemfile
-gem 'drive', '~> 0.2.0'
+gem 'drive', '~> 0.5.0'
 ```
 
-`~> 0.2.0` takes every 0.2.x and stops short of 0.3, which may break what 0.2 does.
+`~> 0.5.0` takes every 0.5.x and stops short of 0.6, which may break what 0.5 does.
 
-Then copy its migration, of runs and changes, and run it:
+Then copy its migrations, of runs, and run them:
 
 ```bash
 bin/rails drive:install:migrations db:migrate
@@ -58,29 +57,16 @@ Run.record 'engines' do
 end
 ```
 
-A table written a record at a time names its key column and where its records come from:
+A table written a record at a time counts each save and delete in the run at hand:
 
 ```ruby
 class Publication < ApplicationRecord
-  include Logged
-
-  KEY = 'document_number'.freeze
-  SOURCE = 'federal_register'.freeze
+  include Counted
 end
 ```
 
-The feed is one line in `config/routes.rb`:
-
-```ruby
-resource :feed, only: :show
-```
-
-Where `FEED_TOKEN` is set, it asks for that token (`Authorization: Bearer …`). Runs commit one
-at a time, so a reader that remembers the last id it applied never misses a change.
-
-The log is meant to hold about a month: call `Change.prune` after each night's refresh, and it
-deletes the changes of runs that started over 30 days ago. A reader that falls further behind
-copies the tables afresh.
+Runs commit one at a time, so another app reading the tables directly sees either the last
+run's rows or the one before's, never a mix.
 
 In tests, `Drive::TestHelper#stage` stages rows as a source's file would:
 

data/app/models/concerns/counted.rb

@@ -0,0 +1,17 @@
+# A table written a record at a time, each save and delete counted in the current run.
+module Counted
+  extend ActiveSupport::Concern
+
+  included do
+    after_save :count_upsert, if: :saved_changes?
+    after_destroy :count_delete
+  end
+
+private
+
+  def count_upsert
+    Run.current.tally(**(previously_new_record? ? { inserted: 1 } : { updated: 1 }))
+  end
+
+  def count_delete = Run.current.tally(deleted: 1)
+end

data/app/models/concerns/mirrored.rb

@@ -17,9 +17,7 @@ module Mirrored
     # @return [Hash{Symbol => Integer}] how many rows were inserted, updated and deleted.
     def mirror(staged, source:)
       stage_incoming staged
-      counts = {
-        inserted: log_inserted(source), updated: log_updated(source), deleted: log_deleted(source),
-      }
+      counts = { inserted: inserted(source), updated: updated(source), deleted: deleted(source) }
       apply source
       counts.tap { Run.current.tally(**it) }
     end
@@ -47,37 +45,30 @@ module Mirrored
       connection.execute 'ANALYZE mirror_incoming'
     end
 
-    def log_inserted(source)
-      log "'upsert'", "to_jsonb(i) - 'row_key' - 'row_hash'", source, <<~SQL.squish
+    def inserted(source)
+      rows <<~SQL.squish
         FROM mirror_incoming i
         WHERE NOT EXISTS (SELECT 1 FROM #{quoted_table_name} t
                           WHERE t.source = #{connection.quote source} AND t.row_key = i.row_key)
       SQL
     end
 
-    def log_updated(source)
-      log "'upsert'", "to_jsonb(i) - 'row_key' - 'row_hash'", source, <<~SQL.squish
+    def updated(source)
+      rows <<~SQL.squish
         FROM mirror_incoming i JOIN #{quoted_table_name} t USING (row_key)
         WHERE t.source = #{connection.quote source} AND t.row_hash <> i.row_hash
       SQL
     end
 
-    def log_deleted(source)
-      log "'delete'", 'NULL', source, <<~SQL.squish
+    def deleted(source)
+      rows <<~SQL.squish
         FROM #{quoted_table_name} i
         WHERE i.source = #{connection.quote source}
           AND NOT EXISTS (SELECT 1 FROM mirror_incoming n WHERE n.row_key = i.row_key)
       SQL
     end
 
-    def log(operation, record, source, from)
-      connection.update <<~SQL.squish
-        INSERT INTO changes (operation, kind, source, row_key, run_id, record, created_at)
-        SELECT #{operation}, #{connection.quote table_name}, #{connection.quote source}, i.row_key,
-               #{Run.current.id}, #{record}, now()
-        #{from}
-      SQL
-    end
+    def rows(from) = connection.select_value("SELECT count(*) #{from}").to_i
 
     def apply(source)
       quoted = fields.map { connection.quote_column_name it }

data/app/models/crawler.rb

@@ -1,33 +1,34 @@
-require 'resolv'
+require 'ipaddr'
+require 'net/http'
 
 # A search engine's own crawler, told apart from a client that only names one in its user agent:
-# its address resolves to the search engine's domain, and that host back to the address, which is
-# how Google and Bing say to check theirs.
+# its address is in the ranges the search engine publishes for it.
 module Crawler
-  # The user agents of the crawlers checked.
-  AGENTS = /Googlebot|Google-InspectionTool|Storebot-Google|GoogleOther|bingbot|Applebot/i
-
-  # The domains their addresses resolve to.
-  DOMAINS = %w[.googlebot.com .google.com .search.msn.com .applebot.apple.com].freeze
+  # The ranges each search engine publishes, by the user agents of its crawlers.
+  RANGES = {
+    /Googlebot|Google-InspectionTool|Storebot-Google|GoogleOther/i =>
+      'https://developers.google.com/static/crawling/ipranges/common-crawlers.json',
+    /bingbot/i => 'https://www.bing.com/toolbox/bingbot.json',
+    /Applebot/i => 'https://search.developer.apple.com/applebot.json',
+    /Claude-SearchBot/i => 'https://claude.com/crawling/bots.json',
+    /OAI-SearchBot/i => 'https://openai.com/searchbot.json',
+    /PerplexityBot/i => 'https://www.perplexity.ai/perplexitybot.json',
+  }.freeze
 
   # @param request [ActionDispatch::Request] request a page was asked with.
-  # @return [Boolean] whether it came from a crawler, checked once a day for each address.
+  # @return [Boolean] whether it came from a crawler, its ranges fetched once a day.
   def self.verified?(request)
-    ip = request.remote_ip
-    request.user_agent.to_s.match?(AGENTS) &&
-      Rails.cache.fetch(['crawler', ip], expires_in: 1.day) { resolves? ip }
+    url = RANGES.find { |agent, _| request.user_agent.to_s.match? agent }&.last
+    url.present? && ranges(url).any? { it.include? request.remote_ip }
   end
 
-  # @param ip [String] address a request came from.
-  # @return [Boolean] whether it names a crawler's host, and that host names it back.
-  def self.resolves?(ip)
-    dns = Resolv::DNS.new.tap { it.timeouts = 2 }
-    host = dns.getname(ip).to_s
-    DOMAINS.any? { host.end_with? it } && dns.getaddresses(host).map(&:to_s).include?(ip)
-  rescue Resolv::ResolvError
-    false
-  ensure
-    dns&.close
+  private_class_method def self.ranges(url)
+    Rails.cache.fetch(['crawler', url], expires_in: 1.day) do
+      uri = URI url
+      body = Net::HTTP.start(uri.host, use_ssl: true, open_timeout: 2, read_timeout: 2) do |http|
+        http.get(uri.path).body
+      end
+      JSON.parse(body)['prefixes'].map { IPAddr.new it.values.first }
+    end
   end
-  private_class_method :resolves?
 end

data/app/models/run.rb

@@ -1,11 +1,8 @@
 # One refresh of a source: when it started and finished, and how many rows it changed.
 class Run < ApplicationRecord
-  # The lock every refresh takes, so runs commit one at a time and their changes in id order.
+  # The lock every refresh takes, so runs commit one at a time.
   LOCK = 1_729_001
 
-  # Named apart from Active Record’s own `changes`, which says what an unsaved record changed.
-  has_many :row_changes, class_name: 'Change', dependent: :delete_all
-
   # Refreshes a source inside one transaction, the only one running, recording what it changed.
   # @param source [String] what is refreshed, such as "registry".
   # @return [Run] finished run.

data/db/migrate/20261006100000_drop_changes.rb

@@ -0,0 +1,16 @@
+class DropChanges < ActiveRecord::Migration[8.1]
+  def change
+    drop_table :changes do |t|
+      t.enum :operation, enum_type: :operation, null: false
+      t.text :kind, null: false
+      t.text :source, null: false
+      t.text :row_key, null: false
+      t.references :run, null: false, foreign_key: true
+      t.jsonb :record
+      t.datetime :created_at, null: false
+      t.index %i[kind row_key]
+    end
+
+    drop_enum :operation, %w[upsert delete]
+  end
+end

data/lib/drive/engine.rb

@@ -1,6 +1,6 @@
 module Drive
-  # Lends the host its models (Run, Change, Mirrored, Logged, AccessFile), the feed's controller
-  # and the migration of runs and changes (`bin/rails drive:install:migrations`).
+  # Lends the host its models (Run, Mirrored, Counted, AccessFile, Crawler), RateLimited and the
+  # migrations of runs (`bin/rails drive:install:migrations`).
   class Engine < ::Rails::Engine
     engine_name 'drive'
   end

data/lib/drive/version.rb

@@ -1,4 +1,4 @@
 module Drive
   # Version of the gem, read by the gemspec.
-  VERSION = '0.3.0'.freeze
+  VERSION = '0.5.0'.freeze
 end

data/lib/drive.rb

@@ -2,6 +2,6 @@ require 'drive/version'
 require 'drive/engine'
 
 # Keeps a copy of the tables another source publishes, changing only the rows the source did,
-# and logs every change for another app to follow (Mirrored, Logged, Run, Change, the feed).
+# and counts what each refresh changed (Mirrored, Counted, Run).
 module Drive
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: drive
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.5.0
 platform: ruby
 authors:
 - claudiob
@@ -52,8 +52,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '8.1'
 description: 'A Rails engine for apps that mirror a public source table for table:
-  each refresh changes only the rows the source did, in one run, and logs each change
-  for another app to follow through a JSON feed.'
+  each refresh changes only the rows the source did, in one run, and counts how many
+  it inserted, updated and deleted.'
 email:
 - claudiob@users.noreply.github.com
 executables: []
@@ -64,15 +64,14 @@ files:
 - MIT-LICENSE
 - README.md
 - app/controllers/concerns/rate_limited.rb
-- app/controllers/feeds_controller.rb
 - app/models/access_file.rb
-- app/models/change.rb
-- app/models/concerns/logged.rb
+- app/models/concerns/counted.rb
 - app/models/concerns/mirrored.rb
 - app/models/crawler.rb
 - app/models/drive/current.rb
 - app/models/run.rb
 - db/migrate/20261005100000_create_runs_and_changes.rb
+- db/migrate/20261006100000_drop_changes.rb
 - lib/drive.rb
 - lib/drive/engine.rb
 - lib/drive/test_helper.rb
@@ -100,5 +99,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 4.0.3
 specification_version: 4
-summary: Keep a copy of what a source publishes, and a feed of every row it changes.
+summary: Keep a copy of what a source publishes, changing only the rows it changed.
 test_files: []

data/app/controllers/feeds_controller.rb

@@ -1,25 +0,0 @@
-# The changes after one another app has read, a page at a time, for it to keep its copy current.
-class FeedsController < ApplicationController
-  # How many changes a page holds.
-  PAGE = 5_000
-
-  # The fields of a change the feed answers.
-  FIELDS = %w[id operation kind source row_key record created_at].freeze
-
-  before_action :authenticate, if: -> { ENV['FEED_TOKEN'].present? }
-
-  # Answers the changes after `after`, of one `kind` (table) or all, as JSON.
-  def show
-    changes = Change.after(params[:after]).limit PAGE
-    changes = changes.where(kind: params[:kind]) if params[:kind]
-    render json: changes.map { it.attributes.slice(*FIELDS) }
-  end
-
-private
-
-  def authenticate
-    authenticate_or_request_with_http_token do |token|
-      ActiveSupport::SecurityUtils.secure_compare token, ENV.fetch('FEED_TOKEN')
-    end
-  end
-end

data/app/models/change.rb

@@ -1,19 +0,0 @@
-# A row a run inserted, changed or deleted, which another app follows by id to keep its copy.
-class Change < ApplicationRecord
-  enum :operation, %w[upsert delete].index_by(&:itself), prefix: true
-
-  belongs_to :run
-
-  # The changes after one another app has already read, in the order they were made.
-  scope :after, ->(id) { where(id: (id.to_i + 1)..).order(:id) }
-
-  # Deletes the changes of runs that started before a time, so the log holds about a month.
-  # @param before [Time] start of the oldest run whose changes are kept.
-  # @return [Integer] changes deleted.
-  def self.prune(before: 30.days.ago)
-    where(run_id: Run.where(started_at: ...before).select(:id)).delete_all
-  end
-
-  # @return [String] operation, table and row key, as the changes' pages name a change.
-  def to_s = "#{operation} #{kind} #{row_key}"
-end

data/app/models/concerns/logged.rb

@@ -1,26 +0,0 @@
-# A table written a record at a time, each save and delete logged as a change of the current run.
-module Logged
-  extend ActiveSupport::Concern
-
-  included do
-    after_save :log_upsert, if: :saved_changes?
-    after_destroy :log_delete
-  end
-
-private
-
-  def log_upsert
-    log :upsert, attributes.except('id', 'created_at', 'updated_at')
-    Run.current.tally(**(previously_new_record? ? { inserted: 1 } : { updated: 1 }))
-  end
-
-  def log_delete
-    log :delete, nil
-    Run.current.tally deleted: 1
-  end
-
-  def log(operation, record)
-    Change.create! operation:, kind: self.class.table_name, source: self.class::SOURCE,
-                   row_key: self[self.class::KEY], run: Run.current, record:
-  end
-end