drive 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a766ae9bb01009e7d8804c036f35fa6b16099c6196d8314d44c6adb166e8460
-  data.tar.gz: 70f366312bbf67d552358624d3ea7cff27da2cbfb45a1dcf3153269c7fb3d7d6
+  metadata.gz: 5bcf5e0361ac44acb57bedda1f651a948abd3ec6bc43cd214080556a142d4b72
+  data.tar.gz: 8c0e23b04b48848ee631f8d08a8f2c97cf15263a4ac5c4c6f58c14654de6fe1a
 SHA512:
-  metadata.gz: 102ef91b1ccbf539b7c0afb3f34e0835ea200ffd0958da6f529045b33af3aef742a495d8a0918b649b95cb2940c38ebec742a7b242d7e434ae2f8f021ba5647a
-  data.tar.gz: 53feb3772e468b24c7c4dc4fb53f3beee4e18cc2c4e152c1dac29e0097221524ee6af4c0718853e29aa469882e2d9413f96f570f52d4c087274494819a99b5b5
+  metadata.gz: 7a671f1e9f52a5a4422cbef14c40ab75d968be2bfe3423b9fccf3c225b6fd0227b5bfe8c301d9acd49036433b4e8efbf389a8b3068492f918aaff9b8139c15bd
+  data.tar.gz: de0351f9c2748aa969ef3fb1e2fbbf027dec38d689b733911f366f4d6e5b1e1f87ec6c7beaf365fa1cd2ab24bbd72f8e7ff42563964e37ebbbabeefd71b780cb

data/CHANGELOG.md

@@ -1,6 +1,16 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+For more information about changelogs, check [Keep a Changelog](http://keepachangelog.com).
+
 ## [Unreleased]
 
-## [0.1.0] - 2026-08-04
+## 0.2.0 - 2026-10-05
+
+* [BREAKING] `drive` is a new gem: a Rails engine for apps that mirror what a source publishes
 
-- Initial release: one `recourses` line in `config/routes.rb` serves a model's
-  index, new and edit screens, each of them overridable by the host app.
+  The name was once a gem of resource screens, which went on as `recourse`; nothing of it is
+  left. This one keeps a copy of a source's tables (`Mirrored`, `Logged`), records each refresh
+  (`Run`) and every row it changed (`Change`), and answers them as a feed (`FeedsController`).
+  `AccessFile` names an Access table's own `ID` column `access_id`, apart from the row's id.

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2026 Claudio Baccigalupo
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,77 +1,97 @@
-# Recourse
+# Drive
 
-A `routes.rb` DSL that mounts ready-made resource screens.
+A Rails engine for an app that keeps a copy of what somebody else publishes — a government's
+files, a monthly database — and tells another app exactly which rows changed.
 
-Add one line to `config/routes.rb` and Recourse draws the routes and serves the
-controllers and views needed to browse a resource. Nothing is written into your
-app — and when you want to customize a screen, you eject it.
+- **`Mirrored`**: a table copying one of the source's tables field for field, as text. A refresh
+  stages the source's table, then inserts, updates and deletes only the rows the source did.
+- **`Logged`**: a table written a record at a time, such as a document fetched by URL.
+- **`Run`**: one refresh, in one transaction, the only one running.
+- **`Change`**: every row a run inserted, updated or deleted, with the row as it is now.
+- **`FeedsController`**: `GET /feed?after=ID&kind=TABLE`, the next 5,000 changes, as JSON.
+- **`AccessFile`**: stages a table of a Microsoft Access file, read with `mdb-export`.
+- **`RateLimited`**: 20 requests a minute and 200 a day per address, for public pages.
 
-> **Status:** early development. `index`, `new`, `create`, `edit` and `update`
-> work; `show`, `destroy` and the eject generator are not implemented yet.
+It needs PostgreSQL.
 
-## Installation
+## How to install
 
-Add the gem to your Gemfile:
+```bash
+gem install drive
+```
 
 ```ruby
-gem 'drive'
+# Gemfile
+gem 'drive', '~> 0.2.0'
+```
+
+`~> 0.2.0` takes every 0.2.x and stops short of 0.3, which may break what 0.2 does.
+
+Then copy its migration, of runs and changes, and run it:
+
+```bash
+bin/rails drive:install:migrations db:migrate
 ```
 
-Then run `bundle install`.
+## How to use
 
-## Usage
+A mirrored table has the source's fields as text columns, plus `source`, `row_key` and
+`row_hash`, unique on `source` and `row_key`. Its model says which fields tell a row apart:
 
 ```ruby
-# config/routes.rb
-Rails.application.routes.draw do
-  recourses :contacts, only: :index
+class Engine < ApplicationRecord
+  include Mirrored
+
+  KEY = %w[code].freeze # [] where the source names none: a row is then told apart by all it says
 end
 ```
 
-With no `ContactsController` and no templates in your app, `/contacts` now lists
-the id of every `Contact`. Recourse supplies both the controller and the view.
+A refresh stages the source's table and mirrors it, inside a run:
+
+```ruby
+Run.record 'engines' do
+  staged = AccessFile.new('ENGINE.accdb').stage('ENGINE', Engine.connection)
+  Engine.mirror staged, source: 'ENGINE'
+end
+```
 
-Anything you write yourself wins. Add `app/controllers/contacts_controller.rb`
-and Recourse leaves it alone; add `app/views/contacts/index.html.erb` and Rails
-renders yours instead of the one the gem ships.
+A table written a record at a time names its key column and where its records come from:
 
-The usual thing to override is a single row. Add
-`app/views/contacts/_row.html.erb` and Recourse's table renders yours for
-`/contacts` while every other resource keeps the default:
+```ruby
+class Publication < ApplicationRecord
+  include Logged
 
-```erb
-<%# locals: (recourse: nil, heading: false) -%>
-<% if heading %>
-  <th scope='col'>Contact</th>
-<% else %>
-  <td data-cell='Contact'><%= recourse.name %></td>
-<% end %>
+  KEY = 'document_number'.freeze
+  SOURCE = 'federal_register'.freeze
+end
 ```
 
-The partial is rendered once for the header row with `heading: true` and no
-record, then once per record with `heading: false`. It builds the cells only —
-the table, the pagination and the layout stay Recourse's.
+The feed is one line in `config/routes.rb`:
 
-Recourse's controllers inherit from your `ApplicationController`, so its pages
-render inside `app/views/layouts/application.html.erb` alongside the rest of your
-app, and go through whatever that base class already does. Each page sets its
-title with `content_for :title`, so put `yield :title` in that layout's `<title>`
-to see it.
+```ruby
+resource :feed, only: :show
+```
 
-## Development
+Where `FEED_TOKEN` is set, it asks for that token (`Authorization: Bearer …`). Runs commit one
+at a time, so a reader that remembers the last id it applied never misses a change.
 
-After checking out the repo, run `bin/setup` to install dependencies. Then run
-`rake test` to run the tests, or `rake` to run the tests and RuboCop. You can
-also run `bin/console` for an interactive prompt.
+The log is meant to hold about a month: call `Change.prune` after each night's refresh, and it
+deletes the changes of runs that started over 30 days ago. A reader that falls further behind
+copies the tables afresh.
 
-To install this gem onto your local machine, run `bundle exec rake install`.
+In tests, `Drive::TestHelper#stage` stages rows as a source's file would:
 
-## Contributing
+```ruby
+ActiveSupport.on_load(:active_support_test_case) { include Drive::TestHelper }
+```
 
-Bug reports and pull requests are welcome on GitHub at
-https://github.com/claudiob/recourse.
+## Development
+
+```bash
+bin/setup        # the bundle, and the dummy app's databases
+bundle exec rake # the suite at 100% line coverage, and RuboCop
+```
 
 ## License
 
-The gem is available as open source under the terms of the
-[MIT License](https://opensource.org/licenses/MIT).
+[MIT](MIT-LICENSE).

data/app/controllers/concerns/rate_limited.rb

@@ -0,0 +1,9 @@
+# The limits of a page anybody may ask for, by the address asking: 20 a minute, 200 a day.
+module RateLimited
+  extend ActiveSupport::Concern
+
+  included do
+    rate_limit to: 20, within: 1.minute, name: 'minute'
+    rate_limit to: 200, within: 1.day, name: 'day'
+  end
+end

data/app/controllers/feeds_controller.rb

@@ -0,0 +1,25 @@
+# The changes after one another app has read, a page at a time, for it to keep its copy current.
+class FeedsController < ApplicationController
+  # How many changes a page holds.
+  PAGE = 5_000
+
+  # The fields of a change the feed answers.
+  FIELDS = %w[id operation kind source row_key record created_at].freeze
+
+  before_action :authenticate, if: -> { ENV['FEED_TOKEN'].present? }
+
+  # Answers the changes after `after`, of one `kind` (table) or all, as JSON.
+  def show
+    changes = Change.after(params[:after]).limit PAGE
+    changes = changes.where(kind: params[:kind]) if params[:kind]
+    render json: changes.map { it.attributes.slice(*FIELDS) }
+  end
+
+private
+
+  def authenticate
+    authenticate_or_request_with_http_token do |token|
+      ActiveSupport::SecurityUtils.secure_compare token, ENV.fetch('FEED_TOKEN')
+    end
+  end
+end

data/app/models/access_file.rb

@@ -0,0 +1,47 @@
+require 'csv'
+
+# A Microsoft Access database, as DRS and the NTSB publish theirs, read with mdb-export (mdbtools).
+class AccessFile
+  # The mdb-export command, writing dates with their century, which it leaves out by default.
+  EXPORT = ['mdb-export', '-D', '%Y-%m-%d', '-T', '%Y-%m-%d %H:%M:%S'].freeze
+
+  # @param path [String] .accdb or .mdb file.
+  def initialize(path)
+    @path = path
+  end
+
+  # Copies a table into a temporary one of text columns, dropped when the transaction commits:
+  # "AD Number" is ad_number, and the table's own id is access_id.
+  # @param table [String] table name in the Access file, such as AD_DATA or events.
+  # @param connection [ActiveRecord::ConnectionAdapters::PostgreSQLAdapter] inside a transaction.
+  # @return [String] temporary table.
+  def stage(table, connection)
+    staged = "staged_#{column table}"
+    IO.popen [*EXPORT, @path, table], 'rb' do |io|
+      columns = CSV.parse_line(io.gets.force_encoding(Encoding::UTF_8)).map { column it }
+      connection.execute "DROP TABLE IF EXISTS #{staged}"
+      connection.execute <<~SQL.squish
+        CREATE TEMP TABLE #{staged} (#{columns.map { %("#{it}" text) }.join ', '}) ON COMMIT DROP
+      SQL
+      copy io, staged, connection.raw_connection
+    end
+    staged
+  end
+
+private
+
+  def column(name)
+    column = name.strip.downcase.gsub(/[^a-z0-9]+/, '_').delete_prefix('_').delete_suffix('_')
+    # An Access table's own counter, named apart from the id Rails gives every row.
+    column == 'id' ? 'access_id' : column
+  end
+
+  def copy(io, staged, raw)
+    raw.copy_data "COPY #{staged} FROM STDIN WITH (FORMAT csv)" do
+      while (chunk = io.read(1 << 20))
+        # Some narratives carry null bytes, which PostgreSQL cannot keep in text.
+        raw.put_copy_data chunk.delete("\x00")
+      end
+    end
+  end
+end

data/app/models/change.rb

@@ -0,0 +1,19 @@
+# A row a run inserted, changed or deleted, which another app follows by id to keep its copy.
+class Change < ApplicationRecord
+  enum :operation, %w[upsert delete].index_by(&:itself), prefix: true
+
+  belongs_to :run
+
+  # The changes after one another app has already read, in the order they were made.
+  scope :after, ->(id) { where(id: (id.to_i + 1)..).order(:id) }
+
+  # Deletes the changes of runs that started before a time, so the log holds about a month.
+  # @param before [Time] start of the oldest run whose changes are kept.
+  # @return [Integer] changes deleted.
+  def self.prune(before: 30.days.ago)
+    where(run_id: Run.where(started_at: ...before).select(:id)).delete_all
+  end
+
+  # @return [String] operation, table and row key, as the changes' pages name a change.
+  def to_s = "#{operation} #{kind} #{row_key}"
+end

data/app/models/concerns/logged.rb

@@ -0,0 +1,26 @@
+# A table written a record at a time, each save and delete logged as a change of the current run.
+module Logged
+  extend ActiveSupport::Concern
+
+  included do
+    after_save :log_upsert, if: :saved_changes?
+    after_destroy :log_delete
+  end
+
+private
+
+  def log_upsert
+    log :upsert, attributes.except('id', 'created_at', 'updated_at')
+    Run.current.tally(**(previously_new_record? ? { inserted: 1 } : { updated: 1 }))
+  end
+
+  def log_delete
+    log :delete, nil
+    Run.current.tally deleted: 1
+  end
+
+  def log(operation, record)
+    Change.create! operation:, kind: self.class.table_name, source: self.class::SOURCE,
+                   row_key: self[self.class::KEY], run: Run.current, record:
+  end
+end

data/app/models/concerns/mirrored.rb

@@ -0,0 +1,99 @@
+# A table copying a source's table field for field as text, changing only the rows the source did.
+module Mirrored
+  extend ActiveSupport::Concern
+
+  # The columns this app adds to what the source published.
+  OWN = %w[id source row_key row_hash created_at updated_at].freeze
+
+  included do
+    # A source's own `type` column is data, not Rails' single-table inheritance.
+    self.inheritance_column = nil
+  end
+
+  class_methods do
+    # Brings the rows from one source in line with a table staged from it, in the current run.
+    # @param staged [String] temporary table holding what the source publishes.
+    # @param source [String] file or document type the rows come from.
+    # @return [Hash{Symbol => Integer}] how many rows were inserted, updated and deleted.
+    def mirror(staged, source:)
+      stage_incoming staged
+      counts = {
+        inserted: log_inserted(source), updated: log_updated(source), deleted: log_deleted(source),
+      }
+      apply source
+      counts.tap { Run.current.tally(**it) }
+    end
+
+  private
+
+    def fields = column_names - OWN
+
+    def stage_incoming(staged)
+      present = connection.columns(staged).map(&:name)
+      picked = fields.map { |field| present.include?(field) ? %(s."#{field}") : 'NULL::text' }
+      hash = "md5(ROW(#{picked.join ', '})::text)"
+      keys = self::KEY.map { %(s."#{it}") }
+      connection.execute 'DROP TABLE IF EXISTS mirror_incoming'
+      connection.execute <<~SQL.squish
+        CREATE TEMP TABLE mirror_incoming ON COMMIT DROP AS
+        SELECT DISTINCT ON (row_key) * FROM (
+          SELECT #{picked.zip(fields).map { |value, field| %(#{value} AS "#{field}") }.join ', '},
+                 #{keys.any? ? "concat_ws('|', #{keys.join ', '})" : hash} AS row_key,
+                 #{hash} AS row_hash
+          FROM #{staged} s
+        ) rows ORDER BY row_key, row_hash
+      SQL
+      connection.execute 'CREATE INDEX ON mirror_incoming (row_key)'
+      connection.execute 'ANALYZE mirror_incoming'
+    end
+
+    def log_inserted(source)
+      log "'upsert'", "to_jsonb(i) - 'row_key' - 'row_hash'", source, <<~SQL.squish
+        FROM mirror_incoming i
+        WHERE NOT EXISTS (SELECT 1 FROM #{quoted_table_name} t
+                          WHERE t.source = #{connection.quote source} AND t.row_key = i.row_key)
+      SQL
+    end
+
+    def log_updated(source)
+      log "'upsert'", "to_jsonb(i) - 'row_key' - 'row_hash'", source, <<~SQL.squish
+        FROM mirror_incoming i JOIN #{quoted_table_name} t USING (row_key)
+        WHERE t.source = #{connection.quote source} AND t.row_hash <> i.row_hash
+      SQL
+    end
+
+    def log_deleted(source)
+      log "'delete'", 'NULL', source, <<~SQL.squish
+        FROM #{quoted_table_name} i
+        WHERE i.source = #{connection.quote source}
+          AND NOT EXISTS (SELECT 1 FROM mirror_incoming n WHERE n.row_key = i.row_key)
+      SQL
+    end
+
+    def log(operation, record, source, from)
+      connection.update <<~SQL.squish
+        INSERT INTO changes (operation, kind, source, row_key, run_id, record, created_at)
+        SELECT #{operation}, #{connection.quote table_name}, #{connection.quote source}, i.row_key,
+               #{Run.current.id}, #{record}, now()
+        #{from}
+      SQL
+    end
+
+    def apply(source)
+      quoted = fields.map { connection.quote_column_name it }
+      connection.delete <<~SQL.squish
+        DELETE FROM #{quoted_table_name} t WHERE t.source = #{connection.quote source}
+        AND NOT EXISTS (SELECT 1 FROM mirror_incoming n WHERE n.row_key = t.row_key)
+      SQL
+      connection.execute <<~SQL.squish
+        INSERT INTO #{quoted_table_name} AS t
+          (source, row_key, row_hash, #{quoted.join ', '}, created_at, updated_at)
+        SELECT #{connection.quote source}, row_key, row_hash, #{quoted.join ', '}, now(), now()
+        FROM mirror_incoming
+        ON CONFLICT (source, row_key) DO UPDATE SET row_hash = excluded.row_hash,
+          #{quoted.map { "#{it} = excluded.#{it}" }.join ', '}, updated_at = now()
+        WHERE t.row_hash <> excluded.row_hash
+      SQL
+    end
+  end
+end

data/app/models/drive/current.rb

@@ -0,0 +1,7 @@
+module Drive
+  # The run of the refresh at hand, which every change belongs to.
+  class Current < ActiveSupport::CurrentAttributes
+    # The run every change of the refresh at hand belongs to.
+    attribute :run
+  end
+end

data/app/models/run.rb

@@ -0,0 +1,36 @@
+# One refresh of a source: when it started and finished, and how many rows it changed.
+class Run < ApplicationRecord
+  # The lock every refresh takes, so runs commit one at a time and their changes in id order.
+  LOCK = 1_729_001
+
+  # Named apart from Active Record’s own `changes`, which says what an unsaved record changed.
+  has_many :row_changes, class_name: 'Change', dependent: :delete_all
+
+  # Refreshes a source inside one transaction, the only one running, recording what it changed.
+  # @param source [String] what is refreshed, such as "registry".
+  # @return [Run] finished run.
+  def self.record(source)
+    transaction do
+      connection.execute "SELECT pg_advisory_xact_lock(#{LOCK})"
+      run = create! source:, started_at: Time.current
+      Drive::Current.set(run:) { yield run }
+      run.tap { it.update! finished_at: Time.current }
+    end
+  end
+
+  # @return [Run] run of the refresh at hand.
+  def self.current = Drive::Current.run
+
+  # Adds what a table's refresh did to this run's counts.
+  # @param inserted [Integer] rows inserted.
+  # @param updated [Integer] rows updated.
+  # @param deleted [Integer] rows deleted.
+  def tally(inserted: 0, updated: 0, deleted: 0)
+    self.inserted += inserted
+    self.updated += updated
+    self.deleted += deleted
+  end
+
+  # @return [String] source and start, as the runs' table names a run.
+  def to_s = "#{source} #{started_at.to_fs :db}"
+end

data/db/migrate/20261005100000_create_runs_and_changes.rb

@@ -0,0 +1,28 @@
+# Each refresh of a source (a run), and every row it inserted, changed or deleted (a change),
+# which is the feed another app follows by the change's id.
+class CreateRunsAndChanges < ActiveRecord::Migration[8.1]
+  def change
+    create_enum :operation, %w[ upsert delete ]
+
+    create_table :runs do |t|
+      t.text :source, null: false
+      t.integer :inserted, default: 0, null: false
+      t.integer :updated, default: 0, null: false
+      t.integer :deleted, default: 0, null: false
+      t.datetime :started_at, null: false
+      t.datetime :finished_at
+      t.virtual :name, type: :text, as: "source || ' #' || id", stored: true
+    end
+
+    create_table :changes do |t|
+      t.enum :operation, enum_type: :operation, null: false
+      t.text :kind, null: false
+      t.text :source, null: false
+      t.text :row_key, null: false
+      t.references :run, null: false, foreign_key: true
+      t.jsonb :record
+      t.datetime :created_at, null: false
+      t.index %i[ kind row_key ]
+    end
+  end
+end

data/lib/drive/engine.rb

@@ -0,0 +1,7 @@
+module Drive
+  # Lends the host its models (Run, Change, Mirrored, Logged, AccessFile), the feed's controller
+  # and the migration of runs and changes (`bin/rails drive:install:migrations`).
+  class Engine < ::Rails::Engine
+    engine_name 'drive'
+  end
+end

data/lib/drive/test_helper.rb

@@ -0,0 +1,20 @@
+module Drive
+  # What a host's tests need to mirror rows without a source's file.
+  module TestHelper
+    # Stages rows as a source's file would, a text column per field, for a model to mirror.
+    # @param table [String] temporary table to create.
+    # @param rows [Array<Hash{String => String}>] the rows, each by field.
+    # @return [String] the table.
+    def stage(table, rows)
+      columns = rows.flat_map(&:keys).uniq
+      connection = ActiveRecord::Base.connection
+      connection.execute "DROP TABLE IF EXISTS #{table}"
+      connection.execute "CREATE TEMP TABLE #{table} (#{columns.map { %("#{it}" text) }.join ', '})"
+      rows.each do |row|
+        values = columns.map { connection.quote row[it] }
+        connection.execute "INSERT INTO #{table} VALUES (#{values.join ', '})"
+      end
+      table
+    end
+  end
+end

data/lib/drive/version.rb

@@ -0,0 +1,4 @@
+module Drive
+  # Version of the gem, read by the gemspec.
+  VERSION = '0.2.0'.freeze
+end

data/lib/drive.rb

@@ -1,3 +1,7 @@
-# Entry point named after the gem, so `require 'drive'` reaches the library that
-# still calls itself Recourse — which is what Bundler does for a gem of this name.
-require_relative 'recourse'
+require 'drive/version'
+require 'drive/engine'
+
+# Keeps a copy of the tables another source publishes, changing only the rows the source did,
+# and logs every change for another app to follow (Mirrored, Logged, Run, Change, the feed).
+module Drive
+end