drive 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5bcf5e0361ac44acb57bedda1f651a948abd3ec6bc43cd214080556a142d4b72
-  data.tar.gz: 8c0e23b04b48848ee631f8d08a8f2c97cf15263a4ac5c4c6f58c14654de6fe1a
+  metadata.gz: cf5ec0cbf839ae0e64fabcecc4f66f40a1e4654252053c57837c8895b290366f
+  data.tar.gz: e5819d0ad8774657ad71eb4695971399ac22215fd7c37d96cf7cc8db3c64f543
 SHA512:
-  metadata.gz: 7a671f1e9f52a5a4422cbef14c40ab75d968be2bfe3423b9fccf3c225b6fd0227b5bfe8c301d9acd49036433b4e8efbf389a8b3068492f918aaff9b8139c15bd
-  data.tar.gz: de0351f9c2748aa969ef3fb1e2fbbf027dec38d689b733911f366f4d6e5b1e1f87ec6c7beaf365fa1cd2ab24bbd72f8e7ff42563964e37ebbbabeefd71b780cb
+  metadata.gz: c2589ac5dc34686550476cf266d06ef8424c0e3d8307761cb4861d15bc3cd7b00884407ac273132f4efa70b4ec1cc671cec67b0bbec70e426aa53d65dbd3bfd3
+  data.tar.gz: e4610a657c780d8458a236bc7512b8dc984e75337e4a17624cff92cc53458799375761027fa5361432c924f0fdfaa088ac5dcde2eddf012039d746ba7e4945f9

data/CHANGELOG.md

@@ -6,6 +6,26 @@ For more information about changelogs, check [Keep a Changelog](http://keepachan
 
 ## [Unreleased]
 
+## 0.4.0 - 2026-10-06
+
+* [BREAKING] No more log of changes, and no feed
+
+  Nothing followed the feed: the apps that read a mirror read its tables directly, from the same
+  database. `Change`, `Change.prune`, `FeedsController` and the `changes` table go;
+  `bin/rails drive:install:migrations db:migrate` copies and runs the migration that drops them.
+  `Mirrored` counts the rows a refresh inserts, updates and deletes instead of logging them, and
+  `Logged` is now `Counted`, which counts each save and delete in the run at hand; a table
+  including it no longer needs `KEY` or `SOURCE`.
+
+## 0.3.0 - 2026-10-06
+
+* [FEATURE] `RateLimited` no longer limits a search engine's own crawler
+
+  `Crawler.verified?(request)` believes a user agent naming Google's, Bing's or Apple's crawler
+  only where its address resolves to their domain and that host back to the address, and
+  remembers the answer for a day. Behind a proxy, `request.remote_ip` must be the visitor's:
+  list the proxy's ranges in `config.action_dispatch.trusted_proxies`.
+
 ## 0.2.0 - 2026-10-05
 
 * [BREAKING] `drive` is a new gem: a Rails engine for apps that mirror what a source publishes

data/README.md

@@ -1,16 +1,18 @@
 # Drive
 
 A Rails engine for an app that keeps a copy of what somebody else publishes — a government's
-files, a monthly database — and tells another app exactly which rows changed.
+files, a monthly database — changing only the rows the source did, and counting them.
 
 - **`Mirrored`**: a table copying one of the source's tables field for field, as text. A refresh
   stages the source's table, then inserts, updates and deletes only the rows the source did.
-- **`Logged`**: a table written a record at a time, such as a document fetched by URL.
-- **`Run`**: one refresh, in one transaction, the only one running.
-- **`Change`**: every row a run inserted, updated or deleted, with the row as it is now.
-- **`FeedsController`**: `GET /feed?after=ID&kind=TABLE`, the next 5,000 changes, as JSON.
+- **`Counted`**: a table written a record at a time, such as a document fetched by URL.
+- **`Run`**: one refresh, in one transaction, the only one running, with how many rows it
+  inserted, updated and deleted.
 - **`AccessFile`**: stages a table of a Microsoft Access file, read with `mdb-export`.
-- **`RateLimited`**: 20 requests a minute and 200 a day per address, for public pages.
+- **`RateLimited`**: 20 requests a minute and 200 a day per address, for public pages. A search
+  engine's own crawler has no limit.
+- **`Crawler`**: whether a request comes from Google's, Bing's or Apple's crawler: its address
+  resolves to their domain and back, checked once a day per address.
 
 It needs PostgreSQL.
 
@@ -22,12 +24,12 @@ gem install drive
 
 ```ruby
 # Gemfile
-gem 'drive', '~> 0.2.0'
+gem 'drive', '~> 0.4.0'
 ```
 
-`~> 0.2.0` takes every 0.2.x and stops short of 0.3, which may break what 0.2 does.
+`~> 0.4.0` takes every 0.4.x and stops short of 0.5, which may break what 0.4 does.
 
-Then copy its migration, of runs and changes, and run it:
+Then copy its migrations, of runs, and run them:
 
 ```bash
 bin/rails drive:install:migrations db:migrate
@@ -55,29 +57,16 @@ Run.record 'engines' do
 end
 ```
 
-A table written a record at a time names its key column and where its records come from:
+A table written a record at a time counts each save and delete in the run at hand:
 
 ```ruby
 class Publication < ApplicationRecord
-  include Logged
-
-  KEY = 'document_number'.freeze
-  SOURCE = 'federal_register'.freeze
+  include Counted
 end
 ```
 
-The feed is one line in `config/routes.rb`:
-
-```ruby
-resource :feed, only: :show
-```
-
-Where `FEED_TOKEN` is set, it asks for that token (`Authorization: Bearer …`). Runs commit one
-at a time, so a reader that remembers the last id it applied never misses a change.
-
-The log is meant to hold about a month: call `Change.prune` after each night's refresh, and it
-deletes the changes of runs that started over 30 days ago. A reader that falls further behind
-copies the tables afresh.
+Runs commit one at a time, so another app reading the tables directly sees either the last
+run's rows or the one before's, never a mix.
 
 In tests, `Drive::TestHelper#stage` stages rows as a source's file would:
 

data/app/controllers/concerns/rate_limited.rb

@@ -1,9 +1,10 @@
-# The limits of a page anybody may ask for, by the address asking: 20 a minute, 200 a day.
+# The limits of a page anybody may ask for, by the address asking: 20 a minute, 200 a day. A search
+# engine's own crawler has none, so it can list every page.
 module RateLimited
   extend ActiveSupport::Concern
 
   included do
-    rate_limit to: 20, within: 1.minute, name: 'minute'
-    rate_limit to: 200, within: 1.day, name: 'day'
+    rate_limit to: 20, within: 1.minute, name: 'minute', unless: -> { Crawler.verified? request }
+    rate_limit to: 200, within: 1.day, name: 'day', unless: -> { Crawler.verified? request }
   end
 end

data/app/models/concerns/counted.rb

@@ -0,0 +1,17 @@
+# A table written a record at a time, each save and delete counted in the current run.
+module Counted
+  extend ActiveSupport::Concern
+
+  included do
+    after_save :count_upsert, if: :saved_changes?
+    after_destroy :count_delete
+  end
+
+private
+
+  def count_upsert
+    Run.current.tally(**(previously_new_record? ? { inserted: 1 } : { updated: 1 }))
+  end
+
+  def count_delete = Run.current.tally(deleted: 1)
+end

data/app/models/concerns/mirrored.rb

@@ -17,9 +17,7 @@ module Mirrored
     # @return [Hash{Symbol => Integer}] how many rows were inserted, updated and deleted.
     def mirror(staged, source:)
       stage_incoming staged
-      counts = {
-        inserted: log_inserted(source), updated: log_updated(source), deleted: log_deleted(source),
-      }
+      counts = { inserted: inserted(source), updated: updated(source), deleted: deleted(source) }
       apply source
       counts.tap { Run.current.tally(**it) }
     end
@@ -47,37 +45,30 @@ module Mirrored
       connection.execute 'ANALYZE mirror_incoming'
     end
 
-    def log_inserted(source)
-      log "'upsert'", "to_jsonb(i) - 'row_key' - 'row_hash'", source, <<~SQL.squish
+    def inserted(source)
+      rows <<~SQL.squish
         FROM mirror_incoming i
         WHERE NOT EXISTS (SELECT 1 FROM #{quoted_table_name} t
                           WHERE t.source = #{connection.quote source} AND t.row_key = i.row_key)
       SQL
     end
 
-    def log_updated(source)
-      log "'upsert'", "to_jsonb(i) - 'row_key' - 'row_hash'", source, <<~SQL.squish
+    def updated(source)
+      rows <<~SQL.squish
         FROM mirror_incoming i JOIN #{quoted_table_name} t USING (row_key)
         WHERE t.source = #{connection.quote source} AND t.row_hash <> i.row_hash
       SQL
     end
 
-    def log_deleted(source)
-      log "'delete'", 'NULL', source, <<~SQL.squish
+    def deleted(source)
+      rows <<~SQL.squish
         FROM #{quoted_table_name} i
         WHERE i.source = #{connection.quote source}
           AND NOT EXISTS (SELECT 1 FROM mirror_incoming n WHERE n.row_key = i.row_key)
       SQL
     end
 
-    def log(operation, record, source, from)
-      connection.update <<~SQL.squish
-        INSERT INTO changes (operation, kind, source, row_key, run_id, record, created_at)
-        SELECT #{operation}, #{connection.quote table_name}, #{connection.quote source}, i.row_key,
-               #{Run.current.id}, #{record}, now()
-        #{from}
-      SQL
-    end
+    def rows(from) = connection.select_value("SELECT count(*) #{from}").to_i
 
     def apply(source)
       quoted = fields.map { connection.quote_column_name it }

data/app/models/crawler.rb

@@ -0,0 +1,33 @@
+require 'resolv'
+
+# A search engine's own crawler, told apart from a client that only names one in its user agent:
+# its address resolves to the search engine's domain, and that host back to the address, which is
+# how Google and Bing say to check theirs.
+module Crawler
+  # The user agents of the crawlers checked.
+  AGENTS = /Googlebot|Google-InspectionTool|Storebot-Google|GoogleOther|bingbot|Applebot/i
+
+  # The domains their addresses resolve to.
+  DOMAINS = %w[.googlebot.com .google.com .search.msn.com .applebot.apple.com].freeze
+
+  # @param request [ActionDispatch::Request] request a page was asked with.
+  # @return [Boolean] whether it came from a crawler, checked once a day for each address.
+  def self.verified?(request)
+    ip = request.remote_ip
+    request.user_agent.to_s.match?(AGENTS) &&
+      Rails.cache.fetch(['crawler', ip], expires_in: 1.day) { resolves? ip }
+  end
+
+  # @param ip [String] address a request came from.
+  # @return [Boolean] whether it names a crawler's host, and that host names it back.
+  def self.resolves?(ip)
+    dns = Resolv::DNS.new.tap { it.timeouts = 2 }
+    host = dns.getname(ip).to_s
+    DOMAINS.any? { host.end_with? it } && dns.getaddresses(host).map(&:to_s).include?(ip)
+  rescue Resolv::ResolvError
+    false
+  ensure
+    dns&.close
+  end
+  private_class_method :resolves?
+end

data/app/models/run.rb

@@ -1,11 +1,8 @@
 # One refresh of a source: when it started and finished, and how many rows it changed.
 class Run < ApplicationRecord
-  # The lock every refresh takes, so runs commit one at a time and their changes in id order.
+  # The lock every refresh takes, so runs commit one at a time.
   LOCK = 1_729_001
 
-  # Named apart from Active Record’s own `changes`, which says what an unsaved record changed.
-  has_many :row_changes, class_name: 'Change', dependent: :delete_all
-
   # Refreshes a source inside one transaction, the only one running, recording what it changed.
   # @param source [String] what is refreshed, such as "registry".
   # @return [Run] finished run.

data/db/migrate/20261006100000_drop_changes.rb

@@ -0,0 +1,16 @@
+class DropChanges < ActiveRecord::Migration[8.1]
+  def change
+    drop_table :changes do |t|
+      t.enum :operation, enum_type: :operation, null: false
+      t.text :kind, null: false
+      t.text :source, null: false
+      t.text :row_key, null: false
+      t.references :run, null: false, foreign_key: true
+      t.jsonb :record
+      t.datetime :created_at, null: false
+      t.index %i[kind row_key]
+    end
+
+    drop_enum :operation, %w[upsert delete]
+  end
+end

data/lib/drive/engine.rb

@@ -1,6 +1,6 @@
 module Drive
-  # Lends the host its models (Run, Change, Mirrored, Logged, AccessFile), the feed's controller
-  # and the migration of runs and changes (`bin/rails drive:install:migrations`).
+  # Lends the host its models (Run, Mirrored, Counted, AccessFile, Crawler), RateLimited and the
+  # migrations of runs (`bin/rails drive:install:migrations`).
   class Engine < ::Rails::Engine
     engine_name 'drive'
   end

data/lib/drive/version.rb

@@ -1,4 +1,4 @@
 module Drive
   # Version of the gem, read by the gemspec.
-  VERSION = '0.2.0'.freeze
+  VERSION = '0.4.0'.freeze
 end

data/lib/drive.rb

@@ -2,6 +2,6 @@ require 'drive/version'
 require 'drive/engine'
 
 # Keeps a copy of the tables another source publishes, changing only the rows the source did,
-# and logs every change for another app to follow (Mirrored, Logged, Run, Change, the feed).
+# and counts what each refresh changed (Mirrored, Counted, Run).
 module Drive
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: drive
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.4.0
 platform: ruby
 authors:
 - claudiob
@@ -52,8 +52,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '8.1'
 description: 'A Rails engine for apps that mirror a public source table for table:
-  each refresh changes only the rows the source did, in one run, and logs each change
-  for another app to follow through a JSON feed.'
+  each refresh changes only the rows the source did, in one run, and counts how many
+  it inserted, updated and deleted.'
 email:
 - claudiob@users.noreply.github.com
 executables: []
@@ -64,14 +64,14 @@ files:
 - MIT-LICENSE
 - README.md
 - app/controllers/concerns/rate_limited.rb
-- app/controllers/feeds_controller.rb
 - app/models/access_file.rb
-- app/models/change.rb
-- app/models/concerns/logged.rb
+- app/models/concerns/counted.rb
 - app/models/concerns/mirrored.rb
+- app/models/crawler.rb
 - app/models/drive/current.rb
 - app/models/run.rb
 - db/migrate/20261005100000_create_runs_and_changes.rb
+- db/migrate/20261006100000_drop_changes.rb
 - lib/drive.rb
 - lib/drive/engine.rb
 - lib/drive/test_helper.rb
@@ -99,5 +99,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 4.0.3
 specification_version: 4
-summary: Keep a copy of what a source publishes, and a feed of every row it changes.
+summary: Keep a copy of what a source publishes, changing only the rows it changed.
 test_files: []

data/app/controllers/feeds_controller.rb

@@ -1,25 +0,0 @@
-# The changes after one another app has read, a page at a time, for it to keep its copy current.
-class FeedsController < ApplicationController
-  # How many changes a page holds.
-  PAGE = 5_000
-
-  # The fields of a change the feed answers.
-  FIELDS = %w[id operation kind source row_key record created_at].freeze
-
-  before_action :authenticate, if: -> { ENV['FEED_TOKEN'].present? }
-
-  # Answers the changes after `after`, of one `kind` (table) or all, as JSON.
-  def show
-    changes = Change.after(params[:after]).limit PAGE
-    changes = changes.where(kind: params[:kind]) if params[:kind]
-    render json: changes.map { it.attributes.slice(*FIELDS) }
-  end
-
-private
-
-  def authenticate
-    authenticate_or_request_with_http_token do |token|
-      ActiveSupport::SecurityUtils.secure_compare token, ENV.fetch('FEED_TOKEN')
-    end
-  end
-end

data/app/models/change.rb

@@ -1,19 +0,0 @@
-# A row a run inserted, changed or deleted, which another app follows by id to keep its copy.
-class Change < ApplicationRecord
-  enum :operation, %w[upsert delete].index_by(&:itself), prefix: true
-
-  belongs_to :run
-
-  # The changes after one another app has already read, in the order they were made.
-  scope :after, ->(id) { where(id: (id.to_i + 1)..).order(:id) }
-
-  # Deletes the changes of runs that started before a time, so the log holds about a month.
-  # @param before [Time] start of the oldest run whose changes are kept.
-  # @return [Integer] changes deleted.
-  def self.prune(before: 30.days.ago)
-    where(run_id: Run.where(started_at: ...before).select(:id)).delete_all
-  end
-
-  # @return [String] operation, table and row key, as the changes' pages name a change.
-  def to_s = "#{operation} #{kind} #{row_key}"
-end

data/app/models/concerns/logged.rb

@@ -1,26 +0,0 @@
-# A table written a record at a time, each save and delete logged as a change of the current run.
-module Logged
-  extend ActiveSupport::Concern
-
-  included do
-    after_save :log_upsert, if: :saved_changes?
-    after_destroy :log_delete
-  end
-
-private
-
-  def log_upsert
-    log :upsert, attributes.except('id', 'created_at', 'updated_at')
-    Run.current.tally(**(previously_new_record? ? { inserted: 1 } : { updated: 1 }))
-  end
-
-  def log_delete
-    log :delete, nil
-    Run.current.tally deleted: 1
-  end
-
-  def log(operation, record)
-    Change.create! operation:, kind: self.class.table_name, source: self.class::SOURCE,
-                   row_key: self[self.class::KEY], run: Run.current, record:
-  end
-end