drive 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d7864c84eb75ad2394534293c1a9f3232eaff9b7a69352d84d8e64605aba56ca
-  data.tar.gz: 6545371dbd8af274b39307dc8139b1cb21ad76f49f4ef15a0ff7478b791eb8fe
+  metadata.gz: ba84684bcb7ceb627421453571e0976616e77b7756ef5677ae79f7837b23945b
+  data.tar.gz: 2068762d2e4c2ec3c94eee70b35cc14b6270d74504c55d02d4be4bc2ccf7b280
 SHA512:
-  metadata.gz: fc337a89d407e9b9ac35ae10febdffb7d202a056bf6403f6533e0cc289965917b74543fa625ee56a15718a9c0ddd818f03fc1fd7cca9cd02e139cf077033aaf9
-  data.tar.gz: 3d3b30536d3427158fd118908f8b9ab98f82e3a876d835cb7f62846c1c250bbda0e9a4c680f890de1c79d8ec30f66eb01a08f5ee13b30afbe1d88498c14af057
+  metadata.gz: 15717e1c2263a8335350fe3d6bcba8d9807df283f6c72180b4e273d6912ef39a1f82172798e7265f189ae92c39b5689ba6cbdf3fd25e08c0be09b1a0aa4b21c0
+  data.tar.gz: 881c0fdbefb2d8466f1443d6d867c1965964b52f68801fd1000cf4117e5c300086f05b1a8cef048558ced13eae2398c4ed38dcb84e4f0cb5663d6e671d5f2541

data/CHANGELOG.md

@@ -6,6 +6,16 @@ For more information about changelogs, check [Keep a Changelog](http://keepachan
 
 ## [Unreleased]
 
+## 0.7.0 - 2026-10-16
+
+* [BREAKING] `Mirrored`, `Counted`, `Run`, `AccessFile` and `Drive::TestHelper` are the `mirroring` gem's
+
+  An app that mirrors adds `gem 'mirroring'` and declares each mirrored model with
+  `mirroring key: :code` (`key: false` where the source names none) in place of `include Mirrored`
+  and `KEY`, and includes `Mirroring::TestHelper` in its tests. `Run.record`, `mirror`,
+  `include Counted` and `AccessFile` keep their names. Drive no longer needs PostgreSQL, nor ships
+  the migrations of runs: an app keeps the `runs` table it has.
+
 ## 0.6.0 - 2026-10-09
 
 * [BREAKING] `include RateLimited` sets no limit by itself: a controller calls `rate_limited`

data/README.md

@@ -1,15 +1,9 @@
 # Drive
 
-A Rails engine for the easyFAA apps. Each keeps a copy of what the FAA or the NTSB publishes,
-changing only the rows the source did and counting them, and each sits behind Cloudflare and
-shows the same error pages.
+A Rails engine for the easyFAA apps. Each sits behind Cloudflare and shows the same error pages.
+A copy of the tables a source publishes is the [`mirroring`](https://github.com/easyfaa/mirroring)
+gem's.
 
-- **`Mirrored`**: a table copying one of the source's tables field for field, as text. A refresh
-  stages the source's table, then inserts, updates and deletes only the rows the source did.
-- **`Counted`**: a table written a record at a time, such as a document fetched by URL.
-- **`Run`**: one refresh, in one transaction, the only one running, with how many rows it
-  inserted, updated and deleted.
-- **`AccessFile`**: stages a table of a Microsoft Access file, read with `mdb-export`.
 - **`RateLimited`**: `rate_limited` holds an address to 20 pages a minute and 200 a day, or the
   app's own limits, and tells it so in words. A search engine's own crawler has no limit.
 - **`Crawler`**: whether a request comes from a search engine's crawler: Cloudflare vouches for
@@ -21,8 +15,6 @@ shows the same error pages.
 - **`NNumber`**, **`Designation`**, **`Serial`**: an N-number, a designation's key and a serial
   number, as every app reads them.
 
-It needs PostgreSQL.
-
 ## How to install
 
 ```bash
@@ -31,55 +23,10 @@ gem install drive
 
 ```ruby
 # Gemfile
-gem 'drive', '~> 0.6.0'
-```
-
-`~> 0.6.0` takes every 0.6.x and stops short of 0.7, which may break what 0.6 does.
-
-Then copy its migrations, of runs, and run them:
-
-```bash
-bin/rails drive:install:migrations db:migrate
-```
-
-## How to use
-
-A mirrored table has the source's fields as text columns, plus `source`, `row_key` and
-`row_hash`, unique on `source` and `row_key`. Its model says which fields tell a row apart:
-
-```ruby
-class Engine < ApplicationRecord
-  include Mirrored
-
-  KEY = %w[code].freeze # [] where the source names none: a row is then told apart by all it says
-end
+gem 'drive', '~> 0.7.0'
 ```
 
-A refresh stages the source's table and mirrors it, inside a run:
-
-```ruby
-Run.record 'engines' do
-  staged = AccessFile.new('ENGINE.accdb').stage('ENGINE', Engine.connection)
-  Engine.mirror staged, source: 'ENGINE'
-end
-```
-
-A table written a record at a time counts each save and delete in the run at hand:
-
-```ruby
-class Publication < ApplicationRecord
-  include Counted
-end
-```
-
-Runs commit one at a time, so another app reading the tables directly sees either the last
-run's rows or the one before's, never a mix.
-
-In tests, `Drive::TestHelper#stage` stages rows as a source's file would:
-
-```ruby
-ActiveSupport.on_load(:active_support_test_case) { include Drive::TestHelper }
-```
+`~> 0.7.0` takes every 0.7.x and stops short of 0.8, which may break what 0.7 does.
 
 ## Development
 

data/lib/drive/engine.rb

@@ -1,7 +1,6 @@
 module Drive
-  # Lends the host its models (Run, Mirrored, Counted, AccessFile, Crawler, NNumber, Designation,
-  # Serial), RateLimited, the error pages and the migrations of runs
-  # (`bin/rails drive:install:migrations`).
+  # Lends the host its models (Crawler, NNumber, Designation, Serial), RateLimited and the error
+  # pages.
   class Engine < ::Rails::Engine
     engine_name 'drive'
 

data/lib/drive/version.rb

@@ -1,4 +1,4 @@
 module Drive
   # Version of the gem, read by the gemspec.
-  VERSION = '0.6.0'.freeze
+  VERSION = '0.7.0'.freeze
 end

data/lib/drive.rb

@@ -3,8 +3,7 @@ require 'drive/inflections'
 require 'drive/cloudflare'
 require 'drive/engine'
 
-# What the easyFAA apps share: a copy of the tables another source publishes, changing only the
-# rows the source did (Mirrored, Counted, Run), and what every app in front of Cloudflare needs
-# alike (Drive::Cloudflare, Crawler, RateLimited, the error pages, NNumber, Designation, Serial).
+# What every easyFAA app in front of Cloudflare needs alike: Drive::Cloudflare, Crawler,
+# RateLimited, the error pages, NNumber, Designation and Serial.
 module Drive
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: drive
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - claudiob
@@ -9,34 +9,6 @@ bindir: bin
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  name: csv
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: pg
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: rails
   requirement: !ruby/object:Gem::Requirement
@@ -51,8 +23,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '8.1'
-description: 'A Rails engine for the easyFAA apps: each mirrors a public source (the
-  FAA, the NTSB) table for table, changing only the rows the source did, and each
+description: 'A Rails engine for the easyFAA apps: each sits behind Cloudflare and
   shows the same error pages, limits, crawlers and N-numbers.'
 email:
 - claudiob@users.noreply.github.com
@@ -65,24 +36,16 @@ files:
 - README.md
 - app/controllers/concerns/rate_limited.rb
 - app/controllers/drive/errors_controller.rb
-- app/models/access_file.rb
-- app/models/concerns/counted.rb
-- app/models/concerns/mirrored.rb
 - app/models/crawler.rb
 - app/models/designation.rb
-- app/models/drive/current.rb
 - app/models/n_number.rb
-- app/models/run.rb
 - app/models/serial.rb
 - app/views/drive/errors/show.html.erb
 - app/views/layouts/drive/errors.html.erb
-- db/migrate/20261005100000_create_runs_and_changes.rb
-- db/migrate/20261006100000_drop_changes.rb
 - lib/drive.rb
 - lib/drive/cloudflare.rb
 - lib/drive/engine.rb
 - lib/drive/inflections.rb
-- lib/drive/test_helper.rb
 - lib/drive/version.rb
 homepage: https://github.com/easyfaa/drive
 licenses:
@@ -105,7 +68,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.3
+rubygems_version: 4.0.20
 specification_version: 4
-summary: 'What the easyFAA apps share: mirrored FAA and NTSB tables, and more.'
+summary: 'What the easyFAA apps share: error pages, Cloudflare, limits and N-numbers.'
 test_files: []

data/app/models/access_file.rb

@@ -1,47 +0,0 @@
-require 'csv'
-
-# A Microsoft Access database, as DRS and the NTSB publish theirs, read with mdb-export (mdbtools).
-class AccessFile
-  # The mdb-export command, writing dates with their century, which it leaves out by default.
-  EXPORT = ['mdb-export', '-D', '%Y-%m-%d', '-T', '%Y-%m-%d %H:%M:%S'].freeze
-
-  # @param path [String] .accdb or .mdb file.
-  def initialize(path)
-    @path = path
-  end
-
-  # Copies a table into a temporary one of text columns, dropped when the transaction commits:
-  # "AD Number" is ad_number, and the table's own id is access_id.
-  # @param table [String] table name in the Access file, such as AD_DATA or events.
-  # @param connection [ActiveRecord::ConnectionAdapters::PostgreSQLAdapter] inside a transaction.
-  # @return [String] temporary table.
-  def stage(table, connection)
-    staged = "staged_#{column table}"
-    IO.popen [*EXPORT, @path, table], 'rb' do |io|
-      columns = CSV.parse_line(io.gets.force_encoding(Encoding::UTF_8)).map { column it }
-      connection.execute "DROP TABLE IF EXISTS #{staged}"
-      connection.execute <<~SQL.squish
-        CREATE TEMP TABLE #{staged} (#{columns.map { %("#{it}" text) }.join ', '}) ON COMMIT DROP
-      SQL
-      copy io, staged, connection.raw_connection
-    end
-    staged
-  end
-
-private
-
-  def column(name)
-    column = name.strip.downcase.gsub(/[^a-z0-9]+/, '_').delete_prefix('_').delete_suffix('_')
-    # An Access table's own counter, named apart from the id Rails gives every row.
-    column == 'id' ? 'access_id' : column
-  end
-
-  def copy(io, staged, raw)
-    raw.copy_data "COPY #{staged} FROM STDIN WITH (FORMAT csv)" do
-      while (chunk = io.read(1 << 20))
-        # Some narratives carry null bytes, which PostgreSQL cannot keep in text.
-        raw.put_copy_data chunk.delete("\x00")
-      end
-    end
-  end
-end

data/app/models/concerns/counted.rb

@@ -1,17 +0,0 @@
-# A table written a record at a time, each save and delete counted in the current run.
-module Counted
-  extend ActiveSupport::Concern
-
-  included do
-    after_save :count_upsert, if: :saved_changes?
-    after_destroy :count_delete
-  end
-
-private
-
-  def count_upsert
-    Run.current.tally(**(previously_new_record? ? { inserted: 1 } : { updated: 1 }))
-  end
-
-  def count_delete = Run.current.tally(deleted: 1)
-end

data/app/models/concerns/mirrored.rb

@@ -1,90 +0,0 @@
-# A table copying a source's table field for field as text, changing only the rows the source did.
-module Mirrored
-  extend ActiveSupport::Concern
-
-  # The columns this app adds to what the source published.
-  OWN = %w[id source row_key row_hash created_at updated_at].freeze
-
-  included do
-    # A source's own `type` column is data, not Rails' single-table inheritance.
-    self.inheritance_column = nil
-  end
-
-  class_methods do
-    # Brings the rows from one source in line with a table staged from it, in the current run.
-    # @param staged [String] temporary table holding what the source publishes.
-    # @param source [String] file or document type the rows come from.
-    # @return [Hash{Symbol => Integer}] how many rows were inserted, updated and deleted.
-    def mirror(staged, source:)
-      stage_incoming staged
-      counts = { inserted: inserted(source), updated: updated(source), deleted: deleted(source) }
-      apply source
-      counts.tap { Run.current.tally(**it) }
-    end
-
-  private
-
-    def fields = column_names - OWN
-
-    def stage_incoming(staged)
-      present = connection.columns(staged).map(&:name)
-      picked = fields.map { |field| present.include?(field) ? %(s."#{field}") : 'NULL::text' }
-      hash = "md5(ROW(#{picked.join ', '})::text)"
-      keys = self::KEY.map { %(s."#{it}") }
-      connection.execute 'DROP TABLE IF EXISTS mirror_incoming'
-      connection.execute <<~SQL.squish
-        CREATE TEMP TABLE mirror_incoming ON COMMIT DROP AS
-        SELECT DISTINCT ON (row_key) * FROM (
-          SELECT #{picked.zip(fields).map { |value, field| %(#{value} AS "#{field}") }.join ', '},
-                 #{keys.any? ? "concat_ws('|', #{keys.join ', '})" : hash} AS row_key,
-                 #{hash} AS row_hash
-          FROM #{staged} s
-        ) rows ORDER BY row_key, row_hash
-      SQL
-      connection.execute 'CREATE INDEX ON mirror_incoming (row_key)'
-      connection.execute 'ANALYZE mirror_incoming'
-    end
-
-    def inserted(source)
-      rows <<~SQL.squish
-        FROM mirror_incoming i
-        WHERE NOT EXISTS (SELECT 1 FROM #{quoted_table_name} t
-                          WHERE t.source = #{connection.quote source} AND t.row_key = i.row_key)
-      SQL
-    end
-
-    def updated(source)
-      rows <<~SQL.squish
-        FROM mirror_incoming i JOIN #{quoted_table_name} t USING (row_key)
-        WHERE t.source = #{connection.quote source} AND t.row_hash <> i.row_hash
-      SQL
-    end
-
-    def deleted(source)
-      rows <<~SQL.squish
-        FROM #{quoted_table_name} i
-        WHERE i.source = #{connection.quote source}
-          AND NOT EXISTS (SELECT 1 FROM mirror_incoming n WHERE n.row_key = i.row_key)
-      SQL
-    end
-
-    def rows(from) = connection.select_value("SELECT count(*) #{from}").to_i
-
-    def apply(source)
-      quoted = fields.map { connection.quote_column_name it }
-      connection.delete <<~SQL.squish
-        DELETE FROM #{quoted_table_name} t WHERE t.source = #{connection.quote source}
-        AND NOT EXISTS (SELECT 1 FROM mirror_incoming n WHERE n.row_key = t.row_key)
-      SQL
-      connection.execute <<~SQL.squish
-        INSERT INTO #{quoted_table_name} AS t
-          (source, row_key, row_hash, #{quoted.join ', '}, created_at, updated_at)
-        SELECT #{connection.quote source}, row_key, row_hash, #{quoted.join ', '}, now(), now()
-        FROM mirror_incoming
-        ON CONFLICT (source, row_key) DO UPDATE SET row_hash = excluded.row_hash,
-          #{quoted.map { "#{it} = excluded.#{it}" }.join ', '}, updated_at = now()
-        WHERE t.row_hash <> excluded.row_hash
-      SQL
-    end
-  end
-end

data/app/models/drive/current.rb

@@ -1,7 +0,0 @@
-module Drive
-  # The run of the refresh at hand, which every change belongs to.
-  class Current < ActiveSupport::CurrentAttributes
-    # The run every change of the refresh at hand belongs to.
-    attribute :run
-  end
-end

data/app/models/run.rb

@@ -1,33 +0,0 @@
-# One refresh of a source: when it started and finished, and how many rows it changed.
-class Run < ApplicationRecord
-  # The lock every refresh takes, so runs commit one at a time.
-  LOCK = 1_729_001
-
-  # Refreshes a source inside one transaction, the only one running, recording what it changed.
-  # @param source [String] what is refreshed, such as "registry".
-  # @return [Run] finished run.
-  def self.record(source)
-    transaction do
-      connection.execute "SELECT pg_advisory_xact_lock(#{LOCK})"
-      run = create! source:, started_at: Time.current
-      Drive::Current.set(run:) { yield run }
-      run.tap { it.update! finished_at: Time.current }
-    end
-  end
-
-  # @return [Run] run of the refresh at hand.
-  def self.current = Drive::Current.run
-
-  # Adds what a table's refresh did to this run's counts.
-  # @param inserted [Integer] rows inserted.
-  # @param updated [Integer] rows updated.
-  # @param deleted [Integer] rows deleted.
-  def tally(inserted: 0, updated: 0, deleted: 0)
-    self.inserted += inserted
-    self.updated += updated
-    self.deleted += deleted
-  end
-
-  # @return [String] source and start, as the runs' table names a run.
-  def to_s = "#{source} #{started_at.to_fs :db}"
-end

data/db/migrate/20261005100000_create_runs_and_changes.rb

@@ -1,28 +0,0 @@
-# Each refresh of a source (a run), and every row it inserted, changed or deleted (a change),
-# which is the feed another app follows by the change's id.
-class CreateRunsAndChanges < ActiveRecord::Migration[8.1]
-  def change
-    create_enum :operation, %w[ upsert delete ]
-
-    create_table :runs do |t|
-      t.text :source, null: false
-      t.integer :inserted, default: 0, null: false
-      t.integer :updated, default: 0, null: false
-      t.integer :deleted, default: 0, null: false
-      t.datetime :started_at, null: false
-      t.datetime :finished_at
-      t.virtual :name, type: :text, as: "source || ' #' || id", stored: true
-    end
-
-    create_table :changes do |t|
-      t.enum :operation, enum_type: :operation, null: false
-      t.text :kind, null: false
-      t.text :source, null: false
-      t.text :row_key, null: false
-      t.references :run, null: false, foreign_key: true
-      t.jsonb :record
-      t.datetime :created_at, null: false
-      t.index %i[ kind row_key ]
-    end
-  end
-end

data/db/migrate/20261006100000_drop_changes.rb

@@ -1,16 +0,0 @@
-class DropChanges < ActiveRecord::Migration[8.1]
-  def change
-    drop_table :changes do |t|
-      t.enum :operation, enum_type: :operation, null: false
-      t.text :kind, null: false
-      t.text :source, null: false
-      t.text :row_key, null: false
-      t.references :run, null: false, foreign_key: true
-      t.jsonb :record
-      t.datetime :created_at, null: false
-      t.index %i[kind row_key]
-    end
-
-    drop_enum :operation, %w[upsert delete]
-  end
-end

data/lib/drive/test_helper.rb

@@ -1,20 +0,0 @@
-module Drive
-  # What a host's tests need to mirror rows without a source's file.
-  module TestHelper
-    # Stages rows as a source's file would, a text column per field, for a model to mirror.
-    # @param table [String] temporary table to create.
-    # @param rows [Array<Hash{String => String}>] the rows, each by field.
-    # @return [String] the table.
-    def stage(table, rows)
-      columns = rows.flat_map(&:keys).uniq
-      connection = ActiveRecord::Base.connection
-      connection.execute "DROP TABLE IF EXISTS #{table}"
-      connection.execute "CREATE TEMP TABLE #{table} (#{columns.map { %("#{it}" text) }.join ', '})"
-      rows.each do |row|
-        values = columns.map { connection.quote row[it] }
-        connection.execute "INSERT INTO #{table} VALUES (#{values.join ', '})"
-      end
-      table
-    end
-  end
-end