mirroring 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0bf6fa454cebfc45cd09e99e1e757558e7aaf3507625d2de1a154800146a2e47
+  data.tar.gz: 77d14645da4d8067e4f542cafa7dd8cacb775b04352e00b3940085cb8db7e293
+SHA512:
+  metadata.gz: cd8b7cb0b7419c2b2edb503af3730024e95ae82ec7cf7dbbdb67b5335ae02bce90029c9b8e4555bcf8de32d93640f4cbe96f5330943f26e3f6ef5981d838f834
+  data.tar.gz: 29b603ff0536109c4601b68aa7f419e0849e8318cf15669357c3fea09787c20fae50b2fb510fc3ca5b3ea2478995aa5e3f60abac0e2a22b4af94f81365c95c57

data/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+For more information about changelogs, check [Keep a Changelog](http://keepachangelog.com).
+
+## [Unreleased]
+
+## 0.1.0 - 2026-10-16
+
+* [FEATURE] `mirroring key:`, `Counted`, `Run` and `AccessFile`, out of drive 0.6
+
+  A model declares `mirroring key: :n_number`, `mirroring key: %i[ev_id aircraft_key]` or
+  `mirroring key: false`, in place of drive's `include Mirrored` and its `KEY` constant. A key
+  that is neither a field, some fields nor `false` raises `ArgumentError` when the class loads.
+  `Run.record`, `Model.mirror`, `include Counted`, `AccessFile#stage` and
+  `Mirroring::TestHelper#stage` work as drive's did. A mirror counts what it inserts, updates and
+  deletes in one query rather than three.
+
+  A host that took the runs' table from drive already has it: it skips the copied
+  `CreateRuns`, or deletes it.

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2026 easyFAA
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,88 @@
+# Mirroring
+
+A Rails engine that keeps a copy of the tables a source publishes, changing only the rows the
+source did and counting them.
+
+- **`mirroring key:`**: a table copying one of the source's tables field for field, as text. A
+  refresh stages the source's table, then inserts, updates and deletes only the rows the source did.
+- **`Counted`**: a table written a record at a time, such as a document fetched by URL.
+- **`Run`**: one refresh, in one transaction, the only one running, with how many rows it
+  inserted, updated and deleted.
+- **`AccessFile`**: stages a table of a Microsoft Access file, read with `mdb-export`.
+
+It needs PostgreSQL.
+
+## How to install
+
+```bash
+gem install mirroring
+```
+
+```ruby
+# Gemfile
+gem 'mirroring', '~> 0.1.0'
+```
+
+`~> 0.1.0` takes every 0.1.x and stops short of 0.2, which may break what 0.1 does.
+
+Then copy its migration, of runs, and run it:
+
+```bash
+bin/rails mirroring:install:migrations db:migrate
+```
+
+## How to use
+
+A mirrored table has the source's fields as text columns, plus `source`, `row_key` and
+`row_hash`, unique on `source` and `row_key`. Its model says which fields tell a row apart:
+
+```ruby
+class Engine < ApplicationRecord
+  mirroring key: :code
+end
+
+class Finding < ApplicationRecord
+  mirroring key: %i[ev_id aircraft_key finding_no]
+end
+
+class State < ApplicationRecord
+  mirroring key: false # the source names no key: a row is told apart by all it says
+end
+```
+
+A refresh stages the source's table and mirrors it, inside a run:
+
+```ruby
+Run.record 'engines' do
+  staged = AccessFile.new('ENGINE.accdb').stage('ENGINE', Engine.connection)
+  Engine.mirror staged, source: 'ENGINE'
+end
+```
+
+A table written a record at a time counts each save and delete in the run at hand:
+
+```ruby
+class Publication < ApplicationRecord
+  include Counted
+end
+```
+
+Runs commit one at a time, so another app reading the tables directly sees either the last
+run's rows or the one before's, never a mix.
+
+In tests, `Mirroring::TestHelper#stage` stages rows as a source's file would:
+
+```ruby
+ActiveSupport.on_load(:active_support_test_case) { include Mirroring::TestHelper }
+```
+
+## Development
+
+```bash
+bin/setup        # the bundle, and the dummy app's test database
+bundle exec rake # the suite at 100% line coverage, RuboCop and file_length
+```
+
+## License
+
+[MIT](MIT-LICENSE).

data/app/models/access_file.rb

@@ -0,0 +1,47 @@
+require 'csv'
+
+# A Microsoft Access database, as DRS and the NTSB publish theirs, read with mdb-export (mdbtools).
+class AccessFile
+  # The mdb-export command, writing dates with their century, which it leaves out by default.
+  EXPORT = ['mdb-export', '-D', '%Y-%m-%d', '-T', '%Y-%m-%d %H:%M:%S'].freeze
+
+  # @param path [String] .accdb or .mdb file.
+  def initialize(path)
+    @path = path
+  end
+
+  # Copies a table into a temporary one of text columns, dropped when the transaction commits:
+  # "AD Number" is ad_number, and the table's own id is access_id.
+  # @param table [String] table name in the Access file, such as AD_DATA or events.
+  # @param connection [ActiveRecord::ConnectionAdapters::PostgreSQLAdapter] inside a transaction.
+  # @return [String] temporary table.
+  def stage(table, connection)
+    staged = "staged_#{column table}"
+    IO.popen [*EXPORT, @path, table], 'rb' do |io|
+      columns = CSV.parse_line(io.gets.force_encoding(Encoding::UTF_8)).map { column it }
+      connection.execute "DROP TABLE IF EXISTS #{staged}"
+      connection.execute <<~SQL.squish
+        CREATE TEMP TABLE #{staged} (#{columns.map { %("#{it}" text) }.join ', '}) ON COMMIT DROP
+      SQL
+      copy io, staged, connection.raw_connection
+    end
+    staged
+  end
+
+private
+
+  def column(name)
+    column = name.strip.downcase.gsub(/[^a-z0-9]+/, '_').delete_prefix('_').delete_suffix('_')
+    # An Access table's own counter, named apart from the id Rails gives every row.
+    column == 'id' ? 'access_id' : column
+  end
+
+  def copy(io, staged, raw)
+    raw.copy_data "COPY #{staged} FROM STDIN WITH (FORMAT csv)" do
+      while (chunk = io.read(1 << 20))
+        # Some narratives carry null bytes, which PostgreSQL cannot keep in text.
+        raw.put_copy_data chunk.delete("\x00")
+      end
+    end
+  end
+end

data/app/models/concerns/counted.rb

@@ -0,0 +1,17 @@
+# A table written a record at a time, each save and delete counted in the current run.
+module Counted
+  extend ActiveSupport::Concern
+
+  included do
+    after_save :count_upsert, if: :saved_changes?
+    after_destroy :count_delete
+  end
+
+private
+
+  def count_upsert
+    Run.current.tally(**(previously_new_record? ? { inserted: 1 } : { updated: 1 }))
+  end
+
+  def count_delete = Run.current.tally(deleted: 1)
+end

data/app/models/concerns/mirrored/incoming.rb

@@ -0,0 +1,50 @@
+# What a source publishes as a mirrored table keeps it, each row with its key and its hash, in a
+# temporary table dropped when the run commits.
+class Mirrored::Incoming
+  # The temporary table, made anew for each table mirrored.
+  TABLE = 'mirror_incoming'
+
+  # Stages the rows of a staged table, the fields a mirrored table has, one row per key.
+  # @param staged [String] temporary table holding what the source publishes.
+  # @param fields [Array<String>] the mirrored table's fields; one the source lacks is NULL.
+  # @param key [Array<String>] fields that tell one row from another; none, and it is all of them.
+  # @param connection [ActiveRecord::ConnectionAdapters::PostgreSQLAdapter] inside a transaction.
+  def initialize(staged, fields:, key:, connection:)
+    @connection = connection
+    present = connection.columns(staged).map(&:name)
+    picked = fields.map { |field| present.include?(field) ? %(s."#{field}") : 'NULL::text' }
+    hash = "md5(ROW(#{picked.join ', '})::text)"
+    keys = key.map { %(s."#{it}") }
+    create <<~SQL.squish
+      SELECT DISTINCT ON (row_key) * FROM (
+        SELECT #{picked.zip(fields).map { |value, field| %(#{value} AS "#{field}") }.join ', '},
+               #{keys.any? ? "concat_ws('|', #{keys.join ', '})" : hash} AS row_key,
+               #{hash} AS row_hash
+        FROM #{staged} s
+      ) rows ORDER BY row_key, row_hash
+    SQL
+  end
+
+  # @param table [String] the mirrored table, quoted.
+  # @param source [String] file or document type the rows come from, quoted.
+  # @return [Hash{Symbol => Integer}] how many of the source's rows the table would insert, update
+  #   and delete to match these.
+  def changes(table:, source:)
+    @connection.select_one(<<~SQL.squish).to_h { |name, count| [name.to_sym, count.to_i] }
+      SELECT count(*) FILTER (WHERE t.row_hash IS NULL) AS inserted,
+             count(*) FILTER (WHERE t.row_hash <> i.row_hash) AS updated,
+             count(*) FILTER (WHERE i.row_hash IS NULL) AS deleted
+      FROM #{TABLE} i
+      FULL JOIN (SELECT row_key, row_hash FROM #{table} WHERE source = #{source}) t USING (row_key)
+    SQL
+  end
+
+private
+
+  def create(rows)
+    @connection.execute "DROP TABLE IF EXISTS #{TABLE}"
+    @connection.execute "CREATE TEMP TABLE #{TABLE} ON COMMIT DROP AS #{rows}"
+    @connection.execute "CREATE INDEX ON #{TABLE} (row_key)"
+    @connection.execute "ANALYZE #{TABLE}"
+  end
+end

data/app/models/concerns/mirrored.rb

@@ -0,0 +1,47 @@
+# A table copying a source's table field for field as text, changing only the rows the source did.
+# A model declares it with `mirroring key:`, which says how rows are told apart.
+module Mirrored
+  extend ActiveSupport::Concern
+
+  # The columns this gem adds to what the source published.
+  OWN = %w[id source row_key row_hash created_at updated_at].freeze
+
+  included do
+    # A source's own `type` column is data, not Rails' single-table inheritance.
+    self.inheritance_column = nil
+  end
+
+  class_methods do
+    # Brings the rows from one source in line with a table staged from it, in the current run.
+    # @param staged [String] temporary table holding what the source publishes.
+    # @param source [String] file or document type the rows come from.
+    # @return [Hash{Symbol => Integer}] how many rows were inserted, updated and deleted.
+    def mirror(staged, source:)
+      incoming = Mirrored::Incoming.new(staged, fields:, key: @mirroring_key, connection:)
+      counts = incoming.changes table: quoted_table_name, source: connection.quote(source)
+      apply source
+      counts.tap { Run.current.tally(**it) }
+    end
+
+  private
+
+    def fields = column_names - OWN
+
+    def apply(source)
+      quoted = fields.map { connection.quote_column_name it }
+      connection.delete <<~SQL.squish
+        DELETE FROM #{quoted_table_name} t WHERE t.source = #{connection.quote source}
+        AND NOT EXISTS (SELECT 1 FROM #{Mirrored::Incoming::TABLE} n WHERE n.row_key = t.row_key)
+      SQL
+      connection.execute <<~SQL.squish
+        INSERT INTO #{quoted_table_name} AS t
+          (source, row_key, row_hash, #{quoted.join ', '}, created_at, updated_at)
+        SELECT #{connection.quote source}, row_key, row_hash, #{quoted.join ', '}, now(), now()
+        FROM #{Mirrored::Incoming::TABLE}
+        ON CONFLICT (source, row_key) DO UPDATE SET row_hash = excluded.row_hash,
+          #{quoted.map { "#{it} = excluded.#{it}" }.join ', '}, updated_at = now()
+        WHERE t.row_hash <> excluded.row_hash
+      SQL
+    end
+  end
+end

data/app/models/mirroring/current.rb

@@ -0,0 +1,7 @@
+module Mirroring
+  # The run of the refresh at hand, which every change belongs to.
+  class Current < ActiveSupport::CurrentAttributes
+    # The run every change of the refresh at hand belongs to.
+    attribute :run
+  end
+end

data/app/models/run.rb

@@ -0,0 +1,33 @@
+# One refresh of a source: when it started and finished, and how many rows it changed.
+class Run < ApplicationRecord
+  # The lock every refresh takes, so runs commit one at a time.
+  LOCK = 1_729_001
+
+  # Refreshes a source inside one transaction, the only one running, recording what it changed.
+  # @param source [String] what is refreshed, such as "registry".
+  # @return [Run] finished run.
+  def self.record(source)
+    transaction do
+      connection.execute "SELECT pg_advisory_xact_lock(#{LOCK})"
+      run = create! source:, started_at: Time.current
+      Mirroring::Current.set(run:) { yield run }
+      run.tap { it.update! finished_at: Time.current }
+    end
+  end
+
+  # @return [Run] run of the refresh at hand.
+  def self.current = Mirroring::Current.run
+
+  # Adds what a table's refresh did to this run's counts.
+  # @param inserted [Integer] rows inserted.
+  # @param updated [Integer] rows updated.
+  # @param deleted [Integer] rows deleted.
+  def tally(inserted: 0, updated: 0, deleted: 0)
+    self.inserted += inserted
+    self.updated += updated
+    self.deleted += deleted
+  end
+
+  # @return [String] source and start, as the runs' table names a run.
+  def to_s = "#{source} #{started_at.to_fs :db}"
+end

data/db/migrate/20261016100000_create_runs.rb

@@ -0,0 +1,14 @@
+# Each refresh of a source (a run): when it started and finished, and how many rows it changed.
+class CreateRuns < ActiveRecord::Migration[8.1]
+  def change
+    create_table :runs do |t|
+      t.text :source, null: false
+      t.integer :inserted, default: 0, null: false
+      t.integer :updated, default: 0, null: false
+      t.integer :deleted, default: 0, null: false
+      t.datetime :started_at, null: false
+      t.datetime :finished_at
+      t.virtual :name, type: :text, as: "source || ' #' || id", stored: true
+    end
+  end
+end

data/lib/mirroring/engine.rb

@@ -0,0 +1,11 @@
+module Mirroring
+  # Lends the host its models (Run, Mirrored, Counted, AccessFile), `mirroring key:` on every model,
+  # and the migration of runs (`bin/rails mirroring:install:migrations`).
+  class Engine < ::Rails::Engine
+    engine_name 'mirroring'
+
+    initializer 'mirroring.macro' do
+      ActiveSupport.on_load(:active_record) { extend Mirroring::Macro }
+    end
+  end
+end

data/lib/mirroring/macro.rb

@@ -0,0 +1,18 @@
+module Mirroring
+  # What every model can declare: `mirroring key: :n_number`.
+  module Macro
+    # Makes the model's table a mirror of a source's (Mirrored), its rows told apart by the key.
+    # @param key [Symbol, String, Array<Symbol, String>, false] field or fields that tell one row
+    #   from another, or false where the source names none: a row is then told apart by all it says.
+    # @raise [ArgumentError] when the key is neither a field, some fields nor false.
+    def mirroring(key:)
+      fields = Array(key)
+      unless key == false || (fields.any? && fields.all? { it in Symbol | String })
+        raise ArgumentError, "key: is a field, some fields or false, not #{key.inspect}"
+      end
+
+      include Mirrored
+      @mirroring_key = key ? fields.map(&:to_s) : []
+    end
+  end
+end

data/lib/mirroring/test_helper.rb

@@ -0,0 +1,20 @@
+module Mirroring
+  # What a host's tests need to mirror rows without a source's file.
+  module TestHelper
+    # Stages rows as a source's file would, a text column per field, for a model to mirror.
+    # @param table [String] temporary table to create.
+    # @param rows [Array<Hash{String => String}>] the rows, each by field.
+    # @return [String] the table.
+    def stage(table, rows)
+      columns = rows.flat_map(&:keys).uniq
+      connection = ActiveRecord::Base.connection
+      connection.execute "DROP TABLE IF EXISTS #{table}"
+      connection.execute "CREATE TEMP TABLE #{table} (#{columns.map { %("#{it}" text) }.join ', '})"
+      rows.each do |row|
+        values = columns.map { connection.quote row[it] }
+        connection.execute "INSERT INTO #{table} VALUES (#{values.join ', '})"
+      end
+      table
+    end
+  end
+end

data/lib/mirroring/version.rb

@@ -0,0 +1,4 @@
+module Mirroring
+  # The released version, as RubyGems lists it.
+  VERSION = '0.1.0'
+end

data/lib/mirroring.rb

@@ -0,0 +1,8 @@
+require 'mirroring/version'
+require 'mirroring/macro'
+require 'mirroring/engine'
+
+# A copy of the tables a source publishes, changing only the rows the source did: `mirroring key:`
+# (Mirrored), Counted and Run.
+module Mirroring
+end

metadata

@@ -0,0 +1,116 @@
+--- !ruby/object:Gem::Specification
+name: mirroring
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- claudiob
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.1'
+- !ruby/object:Gem::Dependency
+  name: csv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: railties
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.1'
+description: 'A Rails engine: a model declared with `mirroring key:` copies a table
+  a source publishes field for field, and each refresh inserts, updates and deletes
+  only the rows the source did, counting them in one run.'
+email:
+- claudiob@users.noreply.github.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- MIT-LICENSE
+- README.md
+- app/models/access_file.rb
+- app/models/concerns/counted.rb
+- app/models/concerns/mirrored.rb
+- app/models/concerns/mirrored/incoming.rb
+- app/models/mirroring/current.rb
+- app/models/run.rb
+- db/migrate/20261016100000_create_runs.rb
+- lib/mirroring.rb
+- lib/mirroring/engine.rb
+- lib/mirroring/macro.rb
+- lib/mirroring/test_helper.rb
+- lib/mirroring/version.rb
+homepage: https://github.com/easyfaa/mirroring
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/easyfaa/mirroring
+  changelog_uri: https://github.com/easyfaa/mirroring/blob/main/CHANGELOG.md
+  documentation_uri: https://rubydoc.info/gems/mirroring
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.4'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.20
+specification_version: 4
+summary: Mirrors a source table for table, changing only the rows the source did.
+test_files: []