deckard 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d27aab47bd5883ad518fe7910ae3faf5f7a3c617430bd0668345dacd652252a6
+  data.tar.gz: 8b232443e73afa31f061a082327c5b3c541912f83127f18ba968e99659f0f167
+SHA512:
+  metadata.gz: e851c6424c4c790437c294461f64e2444a34297c67507b3fa53c36cdc3af2008462ee41f12465908f4187bbd8b02b0ea6ea5cfa9c8f3509d31cbc86de21f69f3
+  data.tar.gz: 354bd7706454a95d042b85d2d80166215cfa0cd9c8ae31c5dfee06f3de0d16f51b277505eaec6a992bdd689ff189044ea536a0220bde49a99484603d2fcd585f

data/CHANGELOG.md

@@ -0,0 +1,58 @@
+# Changelog
+
+All notable changes to deckard are recorded here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow
+[Semantic Versioning](https://semver.org/). Before 1.0, minor versions may
+change the stream format and the Ruby API.
+
+## [0.2.0] - Unreleased
+
+### Changed
+
+- A model's `replicate` block is the plan for dumps rooted at that model,
+  and the only plan those dumps consult. `model "Name" do ... end` entries
+  inside it say how each class the dump reaches is dumped and matched. A
+  reached class's own block is not consulted, so there is nothing to merge
+  and no precedence. Classes are named as strings, and an STI subclass
+  follows its nearest declared ancestor's entry.
+- A record reached only through `belongs_to` carries its row and its own
+  dependencies, nothing it owns: the collections its entry names apply to
+  the root and to records reached through `has_one` or a planned
+  collection. A record dumped first as a dependency and reached later as
+  owned, or as a later root, is walked then and written once.
+- Natural keys travel in the stream. Each replicant is
+  `[type, id, attributes, natural_key]`, the stream version is 2, and a
+  load consults no plan on the destination.
+- Plans are validated before a dump starts: every named class must be a
+  loaded ActiveRecord model and every named association and attribute must
+  exist. Problems are reported together as `ConfigurationError`. The
+  executable eager loads the application (through Zeitwerk, when present)
+  so every `replicate` block has run first.
+- `natural_key` and `omit_fields` store attribute names as strings.
+- When a natural key matches no destination row and the insert that follows
+  fails, `InsertError` names the key that matched nothing.
+
+### Added
+
+- `deckard --plan MODEL [--format text|json]` prints the plan for dumps
+  rooted at a model. `ModelConfig#to_h` builds it as data and
+  `Deckard::PlanReport` renders it.
+- `docs/traversal.md`: how a dump walks the graph, with a worked example.
+- `bin/rspec`.
+
+### Removed
+
+- Per-dump options on `dump` (`associations:`, `omit_fields:`,
+  `omit_associations:`). `dump(object)` and `dump_replicant(dumper)` are the
+  whole API.
+- Per-class configuration inheritance. Plans are keyed by root class name.
+- `Deckard::UnsupportedAssociation`. A `has_and_belongs_to_many` or
+  `has_many :through` association named by a plan is a
+  `ConfigurationError` like any other plan problem.
+
+## [0.1.0]
+
+Initial development version: `-r`/`-d`/`-l` executable, versioned Marshal
+stream, ActiveRecord dumping with automatic `belongs_to` and `has_one`
+traversal, opt-in `has_many`, natural keys, field and association omission,
+custom-object protocol, transactional loads.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Rob Sanheim
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,205 @@
+# Deckard
+
+Deckard copies selected ActiveRecord records, and the records they need,
+from one Rails environment into another. Pick a record, and its plan says
+what comes with it. Rows land with new primary keys, foreign keys are
+rewritten to match, and the load commits only when the stream ends cleanly.
+Its usual job is pulling one production record and its graph into a local
+development database to debug against real data.
+
+It is inspired by [replicate](https://github.com/rtomayko/replicate), rebuilt
+for ActiveRecord 8 and PostgreSQL.
+
+## Quick start
+
+Add the gem to both applications:
+
+```ruby
+gem "deckard"
+```
+
+Put a plan on the model you dump from. It names the collections that come
+along and how each model the dump reaches is matched on the destination:
+
+```ruby
+class Order < ActiveRecord::Base
+  belongs_to :customer
+  has_many :line_items
+
+  replicate do
+    associations :line_items
+
+    model "LineItem" do
+      associations :adjustments
+    end
+
+    model "Customer" do
+      natural_key :email
+      omit_fields :password_digest
+    end
+  end
+end
+```
+
+From your development machine, dump on the remote side and load locally in
+one pipe. SSH is the transport; nothing lands on disk:
+
+```bash
+ssh prod.example.org "deckard -r /app/config/environment -d 'Order.find(1234)'" \
+  | deckard -r ./config/environment -l
+```
+
+Standard error on each end reports what moved:
+
+```text
+loaded 9 total objects:
+
+Adjustment  3
+Customer    1
+LineItem    4
+Order       1
+```
+
+The customer was matched by email, so an existing local customer was updated
+rather than duplicated. Everything else is a new row with a new id.
+
+To keep a dump around, write it to a file and load it later:
+
+```bash
+deckard -r ./config/environment -d "Order.find(1234)" > order.dump
+deckard -r ./config/environment -l < order.dump
+```
+
+## What a dump carries
+
+Dumping a record includes the rows it depends on (`belongs_to`), the record
+itself, the rows it owns one-to-one (`has_one`), and the collections its plan
+names. `has_many` collections are never followed on their own.
+
+A dump follows its root's plan and nothing else: `dump Order.find(1)` uses
+Order's block for every record it reaches, and `dump LineItem.find(1)` uses
+LineItem's. A reached class's own block is not consulted, so there is nothing
+to merge and no precedence to learn.
+
+A record reached only through `belongs_to` is a dependency: it arrives as a
+row with its own dependencies, so foreign keys resolve, and nothing it owns,
+whatever its entry says. The line item's product comes along; the product's
+reviews do not. That rule is what keeps a dump the size of one record's
+ownership tree. [docs/traversal.md](docs/traversal.md) draws it.
+
+Each record handed to `dump` is a root, so `dump Order.where(customer_id: 7)`
+gives every order the full plan, and a record two dumps share lands once.
+
+## Plans
+
+The `replicate` block offers five methods:
+
+```ruby
+replicate do
+  associations :line_items       # opt this has_many into dumps
+  natural_key :number            # reuse an existing destination row
+  omit_fields :internal_notes    # keep a field out of the stream
+  omit_associations :warehouse   # do not traverse this association
+
+  model "LineItem" do            # the same four, for a class the dump reaches
+    associations :adjustments
+  end
+end
+```
+
+Classes are named as strings, so the root never forces them to load first,
+and an STI subclass follows the entry for its nearest declared ancestor. Each
+record travels with the natural key its plan gave it, so the destination
+needs no plan of its own. That also means a natural key is repeated in every
+plan whose dumps reach the class: a Customer entry in Order's plan and one in
+Invoice's, each saying how a customer is matched.
+
+Before dumping, the `deckard` executable eager loads the application so
+every `replicate` block has run, then validates every plan: each named class
+must be a loaded ActiveRecord model, and each named association and attribute
+must exist on it. A bad plan fails there, reporting every problem at once,
+before any record is dumped. Loading consults no plan at all.
+
+To see a plan before pointing it at production, print it:
+
+```bash
+deckard -r ./config/environment --plan Order
+deckard -r ./config/environment --plan Order --format json
+```
+
+```text
+Order
+  associations      line_items
+
+LineItem
+  associations      adjustments
+
+Customer
+  natural key       email
+  omit fields       password_digest
+```
+
+## Dumping
+
+`-d` evaluates a Ruby expression and streams the result to standard output.
+Diagnostics go to standard error; standard output carries only the stream.
+Application boot code that prints to stdout (a logger pointed at `STDOUT`, a
+stray `puts` in an initializer) cannot corrupt it: before requiring the
+application, deckard keeps the original stdout for itself and points file
+descriptor 1 at stderr.
+
+For more involved selection, pass a Ruby file instead of an expression. The
+script runs in a context exposing `dump(object)`:
+
+```ruby
+# config/deckard/dump-repo.rb
+repo = Repository.find_by!(name: ARGV.first)
+dump repo
+dump repo.issues
+```
+
+```bash
+deckard -r ./config/environment -d config/deckard/dump-repo.rb tilt > repos.dump
+```
+
+Extra command-line arguments reach the script through `ARGV`, and `-d -`
+reads the script from standard input. An expression given to `-d` runs in
+the same context, so it may call `dump` directly. While stderr is a
+terminal, a live object counter shows progress on both ends of the pipe.
+
+## Loading
+
+`-l` reads a stream from standard input and loads it inside one transaction,
+bypassing validations and callbacks: it copies data, it does not run your
+application. If the source dies mid-stream, nothing is committed.
+
+Loading refuses to run when the application environment is production. Pass
+`--force` to override.
+
+## Security
+
+The stream is Ruby `Marshal` data: load streams only from applications and
+operators you trust, over an authenticated transport such as SSH. Deckard is
+an internal operator tool, not a public import format. Dumped production data
+lands unmasked in the destination database; treat dumps accordingly. That
+includes ActiveRecord-encrypted attributes: they travel through the stream as
+plaintext and are re-encrypted with the destination's keys on load.
+
+## Development
+
+`bundle exec rake` runs the unit/integration suite (specs against a local
+PostgreSQL 18, lint, and a style ratchet); `bin/rspec spec/deckard/stream_spec.rb`
+runs one file. The specs run against a small forum
+schema managed with ActiveRecord's own migration and schema tooling under
+`spec/db`; `bundle exec rake -T db` lists the database tasks, and
+`bundle exec rake db:reset` rebuilds and seeds the test databases from scratch.
+`bundle exec rake e2e` runs the full-stack test: a real Rails app streaming
+between two docker compose containers. `bundle exec rake test-all` runs both
+host-side checks and the full-stack test. See `docs/spec.md` for the v1.0
+specification and `docs/traversal.md` for how a dump walks the graph.
+
+Crow CI runs `bundle exec rake` on pull requests, default-branch pushes, and
+manual runs using Ruby 4.0.6 and an isolated PostgreSQL 18.6 service. The
+workflow is in `.crow/ruby.yaml`; it covers specs, Standard lint, and the
+style ratchet. The Docker end-to-end harness and performance suite remain
+separate local tasks.

data/exe/deckard

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "deckard"
+require "deckard/cli"
+
+exit Deckard::CLI.new.run(ARGV)

data/lib/deckard/active_record/dump.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+module Deckard
+  module ActiveRecord
+    # Dumps one record by the root's plan (docs/traversal.md), and the records
+    # it reaches by the same plan.
+    class Dump
+      # Stored generated columns (GENERATED ALWAYS AS ... STORED) are never
+      # dumped: the destination database computes them, and PostgreSQL
+      # rejects explicit inserts into them. Computed once per model class.
+      GENERATED_COLUMNS = Hash.new do |cache, model|
+        cache[model] = model.columns.select(&:virtual?).map(&:name)
+      end
+
+      def self.source_id(record)
+        if record.class.primary_key.is_a?(Array)
+          raise DumpError, "#{record.class} has a composite primary key, which deckard does not support"
+        end
+        record.id
+      end
+
+      def initialize(record, dumper, plan, owned:)
+        @record = record
+        @model = record.class
+        @source_id = self.class.source_id(record)
+        @dumper = dumper
+        @plan = plan
+        @owned = owned
+        @entry = plan.for(@model)
+      end
+
+      def call
+        @dumper.visit(@model.name, @source_id, walk: @owned) do
+          attributes = @record.attributes.except(*GENERATED_COLUMNS[@model], *@entry.omitted_fields)
+          dump_belongs_to(attributes)
+          @dumper.write(@model.name, @source_id, attributes, @entry.natural_key_attributes)
+          next unless @owned
+
+          owned = @model.reflect_on_all_associations(:has_one).map(&:name) | @entry.extra_associations
+          (owned - @entry.omitted_associations).each { |name| dump_owned(@record.public_send(name)) }
+        end
+      end
+
+      private
+
+      def dump_belongs_to(attributes)
+        @model.reflect_on_all_associations(:belongs_to).each do |reflection|
+          foreign_key = reflection.foreign_key.to_s
+          next if @entry.omitted_associations.include?(reflection.name)
+          next if encode_dumped_parent(attributes, reflection, foreign_key)
+
+          referenced = @record.public_send(reflection.name)
+          next if referenced.nil?
+
+          self.class.new(referenced, @dumper, @plan, owned: false).call
+          referenced_id = self.class.source_id(referenced)
+          unless @dumper.dumped?(referenced.class.name, referenced_id)
+            raise DumpError,
+              "dependency cycle detected: #{@model}(#{@source_id}).#{reflection.name} " \
+              "references #{referenced.class.name}(#{referenced_id}), which cannot be emitted first"
+          end
+          next if @entry.omitted_fields.include?(foreign_key)
+          # A belongs_to whose primary_key option targets a non-primary-key
+          # column (belongs_to :account, primary_key: :login) carries a natural
+          # value, not a source ID. It needs no remapping and must not be
+          # replaced with a destination primary key.
+          next unless reflection.association_primary_key(referenced.class) == referenced.class.primary_key
+
+          attributes[foreign_key] = [:id, referenced.class.name, referenced_id]
+        end
+      end
+
+      # When the parent is already in the stream under the association's
+      # declared class, encode the reference from the foreign key alone
+      # instead of loading the parent again. A parent dumped as an STI
+      # subclass misses this check and takes the loading path.
+      def encode_dumped_parent(attributes, reflection, foreign_key)
+        return false if reflection.polymorphic?
+        return false unless reflection.association_primary_key == reflection.klass.primary_key
+
+        value = @record[foreign_key]
+        return false if value.nil? || !@dumper.dumped?(reflection.klass.name, value)
+
+        attributes[foreign_key] = [:id, reflection.klass.name, value] unless @entry.omitted_fields.include?(foreign_key)
+        true
+      end
+
+      def dump_owned(associated)
+        records = associated.respond_to?(:find_each) ? associated.find_each : Array(associated)
+        records.each { |record| self.class.new(record, @dumper, @plan, owned: true).call }
+      end
+    end
+  end
+end

data/lib/deckard/active_record/load.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+module Deckard
+  module ActiveRecord
+    # Loads one replicant tuple into a model's table.
+    class Load
+      def initialize(model, type, source_id, attributes, natural_key)
+        @model = model
+        @type = type
+        @source_id = source_id
+        @attributes = attributes
+        @natural_key = natural_key
+      end
+
+      def call
+        if @model.primary_key.is_a?(Array)
+          raise LoadError, "#{@model.name} has a composite primary key, which deckard does not support"
+        end
+
+        @natural_key.empty? ? insert : load_by_natural_key
+      end
+
+      private
+
+      # `because` says what led to the insert when it was not the only path,
+      # so a failure names the natural key that matched nothing.
+      def insert(because = "")
+        result = @model.insert_all!([@attributes.except(@model.primary_key)], returning: [@model.primary_key])
+        destination_id = result.rows.first.first
+        [destination_id, @model.find(destination_id)]
+      rescue ::ActiveRecord::ActiveRecordError => e
+        raise InsertError,
+          "#{@type} source_id=#{@source_id} #{because}could not be inserted: #{Deckard.error_detail(e)}"
+      end
+
+      def load_by_natural_key
+        lookup = @attributes.slice(*@natural_key)
+        matches = @model.where(lookup).limit(2).to_a
+
+        case matches.size
+        when 0
+          insert("matched no destination row by natural key (#{lookup.keys.join(", ")}) and ")
+        when 1
+          record = matches.first
+          begin
+            record.update_columns(@attributes.except(@model.primary_key))
+          rescue ::ActiveRecord::ActiveRecordError => e
+            raise InsertError,
+              "#{@type} source_id=#{@source_id} could not be updated via natural key: #{Deckard.error_detail(e)}"
+          end
+          [record.id, record]
+        else
+          raise LoadError,
+            "#{@type} source_id=#{@source_id}: natural key (#{lookup.keys.join(", ")}) matches more than one destination row"
+        end
+      end
+    end
+  end
+end

data/lib/deckard/active_record.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module Deckard
+  # Implements the replicant protocol for ActiveRecord models. Included into
+  # ActiveRecord::Base when ActiveRecord loads (see lib/deckard.rb). Models
+  # gain exactly three methods: the `replicate` configuration DSL,
+  # `dump_replicant`, and `load_replicant`. Traversal and loading live in
+  # the plain objects Dump (lib/deckard/active_record/dump.rb) and Load
+  # (lib/deckard/active_record/load.rb) so nothing else lands in the model
+  # namespace.
+  module ActiveRecord
+    def self.included(base)
+      base.extend ClassMethods
+    end
+
+    # Records reached from this one are dumped by its plan directly, so an
+    # override of this method applies to roots only.
+    def dump_replicant(dumper)
+      Dump.new(self, dumper, ModelConfig.plan_for(self.class), owned: true).call
+    end
+
+    module ClassMethods
+      # The `replicate do ... end` DSL: the plan for dumps rooted here.
+      def replicate(&block)
+        ModelConfig.declare(name, &block)
+      end
+
+      # Load one streamed replicant: reuse an existing row when its natural
+      # key matches, otherwise insert a new row with a destination-generated
+      # primary key. Both paths bypass validations and callbacks.
+      def load_replicant(type, source_id, attributes, natural_key)
+        Load.new(self, type, source_id, attributes, natural_key).call
+      end
+    end
+  end
+end

data/lib/deckard/cli.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+require "optimist"
+require_relative "plan_report"
+require_relative "status"
+
+module Deckard
+  # The deckard executable: -r requires the application environment, -d
+  # dumps a Ruby expression or dump script to stdout, -l loads a stream
+  # from stdin, --plan prints a root model's plan. Required from
+  # exe/deckard, not from the library itself.
+  class CLI
+    # Evaluation context for dump scripts and -d expressions: exposes only
+    # dump(object). Scripts see any extra command-line arguments in ARGV.
+    class DumpScript
+      def initialize(dumper)
+        @dumper = dumper
+      end
+
+      def dump(object)
+        @dumper.dump(object)
+      end
+    end
+
+    def initialize
+      @stdin = $stdin
+      @stdout = $stdout
+      @stderr = $stderr
+    end
+
+    def run(argv)
+      options = parse(argv)
+      reserve_stdout if options[:dump]
+      require File.expand_path(options[:require]) if options[:require]
+
+      if options[:dump]
+        dump(options[:dump])
+      elsif options[:plan]
+        print_plan(options[:plan], options[:format])
+      else
+        load_stream(options)
+      end
+      0
+    rescue Error => e
+      @stderr.puts "#{e.class}: #{e.message}"
+      1
+    rescue Errno::EPIPE
+      @stderr.puts "deckard: output pipe closed before the stream completed"
+      1
+    end
+
+    private
+
+    # Application boot code may print to stdout, which would corrupt the
+    # stream. Keep a private copy of the original stdout for the dumper and
+    # point file descriptor 1 at stderr, so anything else this process or
+    # its children write to stdout lands on stderr instead.
+    # rubocop:disable Style/GlobalStdStream -- the process-level streams are the point
+    def reserve_stdout
+      @stdout = STDOUT.dup
+      STDOUT.reopen(STDERR)
+      $stdout = STDOUT
+    end
+    # rubocop:enable Style/GlobalStdStream
+
+    def parse(argv)
+      options = Optimist.options(argv) do
+        version "deckard #{VERSION}"
+        banner <<~BANNER
+          deckard: stream ActiveRecord objects between Rails environments
+
+            dump:  deckard -r ./config/environment -d "User.find(1)" > user.dump
+            load:  deckard -r ./config/environment -l < user.dump
+            pipe:  ssh example.org "deckard -r /app/config/environment -d 'User.find(1)'" | deckard -r ./config/environment -l
+            plan:  deckard -r ./config/environment --plan User
+
+          Options:
+        BANNER
+        opt :require, "Ruby file to require first (usually config/environment)", type: :string
+        opt :dump, "Dump the result of a Ruby expression, or run a dump script (a file, or - for stdin)", type: :string
+        opt :load, "Load a deckard stream from standard input"
+        opt :plan, "Print the plan for dumps rooted at MODEL", type: :string
+        opt :format, "Plan output: #{PlanReport::FORMATS.join(" or ")}", default: "text"
+        opt :force, "Allow loading into a production environment"
+      end
+
+      unless [options[:dump], options[:plan], options[:load] || nil].compact.size == 1
+        Optimist.die "exactly one of -d, -l or --plan is required"
+      end
+      Optimist.die :format, "must be one of #{PlanReport::FORMATS.join(", ")}" unless PlanReport::FORMATS.include?(options[:format])
+      options
+    end
+
+    # Every replicate block must have run before a plan is read, and Rails
+    # only eager loads on its own when config.eager_load is on.
+    def eager_load
+      Zeitwerk::Loader.eager_load_all if defined?(Zeitwerk::Loader)
+    end
+
+    def dump(target)
+      eager_load
+      ModelConfig.validate!
+      @stdout.binmode
+      dumper = Dumper.new(@stdout) { |counts| Status.progress("dumping", counts, @stderr) }
+      script = DumpScript.new(dumper)
+
+      if target == "-"
+        script.instance_eval(@stdin.read, "<stdin>")
+      elsif File.exist?(target)
+        script.instance_eval(File.read(target), target)
+      else
+        dumper.dump(script.instance_eval(target, "deckard -d"))
+      end
+
+      dumper.complete
+      @stdout.flush
+      Status.report("dumped", dumper.counts, @stderr)
+    end
+
+    def print_plan(model_name, format)
+      eager_load
+      klass = model_name.safe_constantize
+      raise ConfigurationError, "#{model_name.inspect} is not a loaded ActiveRecord model" unless klass.respond_to?(:replicate)
+
+      plan = ModelConfig.plan_for(klass)
+      raise ConfigurationError, "#{klass} has no replicate block" if plan.equal?(ModelConfig::NONE)
+
+      problems = plan.problems
+      raise ConfigurationError, problems.join("\n") unless problems.empty?
+
+      @stdout.write PlanReport.render(plan.to_h, format)
+    end
+
+    def load_stream(options)
+      if Deckard.production_environment? && !options[:force]
+        raise LoadError, "refusing to load into a production environment (pass --force to override)"
+      end
+
+      @stdin.binmode
+      loader = Loader.new(@stdin) { |counts| Status.progress("loading", counts, @stderr) }
+      loader.load
+      Status.report("loaded", loader.counts, @stderr)
+    end
+  end
+end

data/lib/deckard/dumper.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+module Deckard
+  # Writes a versioned Marshal stream of replicant tuples to an output IO.
+  # Objects are deduplicated by [type, source_id]: a repeated write is a no-op.
+  class Dumper
+    attr_reader :counts
+
+    # An optional block is called with the running counts after every
+    # written replicant; the CLI uses it for a progress line.
+    def initialize(output, &after_write)
+      @output = output
+      @after_write = after_write
+      @dumped = Set.new
+      @walked = Set.new
+      @in_progress = Set.new
+      @counts = Hash.new(0)
+      write_frame(STREAM_HEADER)
+    end
+
+    # Dump one object, or each element of an enumerable. The object must
+    # implement dump_replicant(dumper) and write itself (and any
+    # dependencies, first) via #write.
+    def dump(object)
+      return if object.nil?
+
+      if object.respond_to?(:dump_replicant)
+        object.dump_replicant(self)
+      elsif object.respond_to?(:find_each)
+        object.find_each { |item| dump(item) }
+      elsif object.respond_to?(:each)
+        object.each { |item| dump(item) }
+      else
+        raise DumpError, "#{object.class} does not implement dump_replicant"
+      end
+    end
+
+    # Runs the block for a [type, id] not yet written, or written but not
+    # yet walked when walk is true. A record visited again while its own
+    # visit is in progress is a dependency cycle: the block is skipped, and
+    # callers that need the identity written first check #dumped? and raise.
+    def visit(type, id, walk:)
+      key = [type, id]
+      return if @in_progress.include?(key) || (walk ? @walked : @dumped).include?(key)
+
+      @walked.add(key) if walk
+      @in_progress.add(key)
+      begin
+        yield
+      ensure
+        @in_progress.delete(key)
+      end
+    end
+
+    def dumped?(type, id)
+      @dumped.include?([type, id])
+    end
+
+    # Called by dump_replicant implementations to emit one replicant tuple.
+    # natural_key names the attributes by which the destination matches an
+    # existing record to reuse; empty means always insert.
+    def write(type, id, attributes, natural_key = [])
+      type = type.to_s
+      return unless @dumped.add?([type, id])
+
+      write_frame([type, id, attributes, natural_key])
+      @counts[type] += 1
+      @after_write&.call(@counts)
+    end
+
+    # Write the successful-end marker. A stream without it is treated as
+    # truncated and never commits on the destination.
+    def complete
+      write_frame(STREAM_END)
+    end
+
+    private
+
+    def write_frame(frame)
+      Marshal.dump(frame, @output)
+    rescue Errno::EPIPE
+      raise
+    rescue IOError, SystemCallError => e
+      raise OutputError, "could not write dump stream: #{Deckard.error_detail(e)}", cause: e
+    end
+  end
+end

data/lib/deckard/errors.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Deckard
+  def self.error_detail(error)
+    detail = error.message.to_s.lines.first.to_s.strip
+    detail.empty? ? "(no message)" : detail
+  end
+
+  class Error < StandardError; end
+
+  class ConfigurationError < Error; end
+
+  class DumpError < Error; end
+
+  class OutputError < DumpError; end
+
+  class LoadError < Error; end
+
+  class UnresolvedReference < LoadError; end
+
+  class InvalidStream < LoadError; end
+
+  class InsertError < LoadError; end
+end

data/lib/deckard/loader.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+module Deckard
+  # Reads a Deckard stream incrementally from an input IO, resolving
+  # [:id, type, source_id] reference tuples through the source-to-destination
+  # ID map and handing each replicant, with the natural key it travels
+  # with, to its class's load_replicant.
+  class Loader
+    attr_reader :counts
+
+    # An optional block is called with the running counts after every
+    # loaded replicant; the CLI uses it for a progress line.
+    def initialize(input, &after_load)
+      @input = input
+      @after_load = after_load
+      @id_map = {}
+      @counts = Hash.new(0)
+    end
+
+    def load
+      with_transaction do
+        header = read_frame
+        unless header == STREAM_HEADER
+          raise InvalidStream, "invalid stream header (expected #{STREAM_HEADER.inspect})"
+        end
+
+        loop do
+          frame = read_frame
+          break if frame == STREAM_END
+          load_replicant(frame)
+        end
+      end
+    end
+
+    private
+
+    # The whole load runs in one destination transaction, committing only
+    # after a valid end marker. Without a database connection (custom-object
+    # streams outside a booted app) the load runs bare.
+    def with_transaction(&block)
+      return yield unless defined?(::ActiveRecord::Base)
+
+      begin
+        ::ActiveRecord::Base.connection_pool
+      rescue ::ActiveRecord::ConnectionNotEstablished
+        return yield
+      end
+      ::ActiveRecord::Base.transaction(&block)
+    end
+
+    def read_frame
+      Marshal.load(@input)
+    rescue EOFError
+      raise InvalidStream, "stream ended without an end marker"
+    rescue TypeError, ArgumentError => e
+      raise InvalidStream, "corrupt stream frame: #{e.message}"
+    end
+
+    def load_replicant(frame)
+      unless frame.is_a?(Array) && frame.size == 4 && frame[0].is_a?(String) && frame[2].is_a?(Hash) && frame[3].is_a?(Array)
+        raise InvalidStream, "malformed stream frame (expected [type, id, attributes, natural_key] tuple)"
+      end
+
+      type, source_id, attributes, natural_key = frame
+      resolved = resolve_references(type, source_id, attributes)
+      destination_id, _object = replicant_class(type).load_replicant(type, source_id, resolved, natural_key)
+      @id_map[[type, source_id]] = destination_id
+      @counts[type] += 1
+      @after_load&.call(@counts)
+    end
+
+    def resolve_references(type, source_id, attributes)
+      attributes.each_with_object({}) do |(name, value), resolved|
+        resolved[name] = reference?(value) ? resolve_reference(type, source_id, name, value) : value
+      end
+    end
+
+    def reference?(value)
+      value.is_a?(Array) && value.size == 3 && value[0] == :id
+    end
+
+    def resolve_reference(type, source_id, name, reference)
+      _, ref_type, ref_id = reference
+      @id_map.fetch([ref_type, ref_id]) do
+        raise UnresolvedReference,
+          "#{type}(#{source_id}).#{name} references #{ref_type}(#{ref_id}), which has not been loaded"
+      end
+    end
+
+    def replicant_class(type)
+      klass = begin
+        Object.const_get(type)
+      rescue NameError
+        raise LoadError, "cannot load #{type}: class is not defined"
+      end
+
+      unless klass.respond_to?(:load_replicant)
+        raise LoadError, "cannot load #{type}: #{klass} does not implement load_replicant"
+      end
+      klass
+    end
+  end
+end

data/lib/deckard/model_config.rb

@@ -0,0 +1,129 @@
+# frozen_string_literal: true
+
+require "active_support/core_ext/string/inflections"
+
+module Deckard
+  # A replication plan: a root model's `replicate do ... end` block. The plan
+  # is an entry for the root plus, through `model`, an entry for each class
+  # its dumps reach, all in one table shared by the plan's entries. A class
+  # finds its entry through its ancestors, so an STI subclass follows its
+  # nearest declared ancestor. See docs/traversal.md.
+  class ModelConfig
+    attr_reader :name, :extra_associations, :natural_key_attributes, :omitted_fields, :omitted_associations
+
+    @plans = {}
+
+    class << self
+      def declare(name, &block)
+        @plans[name] = new(name).tap { |plan| plan.instance_eval(&block) }
+      end
+
+      def plan_for(klass)
+        @plans.values_at(*klass.ancestors.map(&:name)).compact.first || NONE
+      end
+
+      # Every problem in every plan, so one run fixes them all.
+      def validate!
+        problems = @plans.each_value.flat_map(&:problems)
+        raise ConfigurationError, problems.join("\n") unless problems.empty?
+      end
+    end
+
+    def initialize(name = nil, entries = {})
+      @name = name
+      @entries = entries
+      @entries[name] = self if name
+      @resolved = {}
+      @extra_associations = []
+      @natural_key_attributes = []
+      @omitted_fields = []
+      @omitted_associations = []
+    end
+
+    # DSL: association names to dump beyond the automatic belongs_to and
+    # has_one traversal. Additive across calls.
+    def associations(*names)
+      @extra_associations |= names.map(&:to_sym)
+    end
+
+    # DSL: attributes identifying an existing destination record to reuse.
+    # Calling it again replaces the key.
+    def natural_key(*attributes)
+      @natural_key_attributes = attributes.map(&:to_s)
+    end
+
+    # DSL: fields to exclude from dumped attributes. Additive across calls.
+    def omit_fields(*names)
+      @omitted_fields |= names.map(&:to_s)
+    end
+
+    # DSL: associations not to traverse. Additive across calls.
+    def omit_associations(*names)
+      @omitted_associations |= names.map(&:to_sym)
+    end
+
+    # DSL: the entry for a class this plan's dumps reach.
+    def model(name, &block)
+      (@entries[name.to_s] ||= self.class.new(name.to_s, @entries)).instance_eval(&block)
+    end
+
+    # The entry that applies to klass, checked against it the first time.
+    def for(klass)
+      @resolved[klass] ||= (@entries.values_at(*klass.ancestors.map(&:name)).compact.first || NONE).tap do |entry|
+        problems = entry.problems_for(klass)
+        raise ConfigurationError, problems.join("\n") unless problems.empty?
+      end
+    end
+
+    def problems
+      @entries.each_value.flat_map do |entry|
+        klass = entry.name.safe_constantize
+        next ["#{entry.name.inspect} is named in a replicate block, but is not a loaded ActiveRecord model"] unless klass.respond_to?(:replicate)
+
+        entry.problems_for(klass)
+      end
+    end
+
+    # Everything this entry names that the model lacks or deckard cannot
+    # traverse.
+    def problems_for(model)
+      problems = @extra_associations.filter_map do |name|
+        reflection = model.reflect_on_association(name)
+        if reflection.nil?
+          "#{model} has no #{name.inspect} association"
+        elsif reflection.macro == :has_and_belongs_to_many
+          "#{model}.#{name} is a has_and_belongs_to_many association, " \
+            "which deckard does not support; use an explicit join model and replicate that association instead"
+        elsif reflection.macro == :has_many && reflection.through_reflection
+          "#{model}.#{name} is a has_many :through association, " \
+            "which deckard does not support; replicate :#{reflection.through_reflection.name} instead"
+        end
+      end
+      attributes = @natural_key_attributes + @omitted_fields
+      unless attributes.empty? || model.abstract_class?
+        missing = attributes - model.attribute_names
+        problems += missing.map { |attribute| "#{model} has no #{attribute.inspect} attribute" }
+      end
+      problems + (@natural_key_attributes & @omitted_fields).map do |attribute|
+        "#{model} names #{attribute.inspect} as both a natural key attribute and an omitted field"
+      end
+    end
+
+    # The plan as data, root entry first: what PlanReport renders.
+    def to_h
+      entries = @entries.each_value.map do |entry|
+        {
+          "model" => entry.name,
+          "associations" => entry.extra_associations.map(&:to_s),
+          "natural_key" => entry.natural_key_attributes,
+          "omit_fields" => entry.omitted_fields,
+          "omit_associations" => entry.omitted_associations.map(&:to_s)
+        }
+      end
+      {"root" => @name, "entries" => entries}
+    end
+
+    # The entry for a class nothing declared: defaults only.
+    NONE = new.freeze
+  end
+end

data/lib/deckard/plan_report.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Deckard
+  # Renders a plan's #to_h for people (text) or tools (json). Rendering
+  # never looks at the plan itself, so a new format is a new method here.
+  module PlanReport
+    FORMATS = %w[text json].freeze
+
+    LINES = {
+      "associations" => "associations",
+      "natural_key" => "natural key",
+      "omit_fields" => "omit fields",
+      "omit_associations" => "omit associations"
+    }.freeze
+
+    def self.render(plan, format)
+      (format == "json") ? "#{JSON.pretty_generate(plan)}\n" : text(plan)
+    end
+
+    def self.text(plan)
+      blocks = plan["entries"].map do |entry|
+        lines = LINES.filter_map do |key, label|
+          "  #{label.ljust(18)}#{entry[key].join(", ")}" unless entry[key].empty?
+        end
+        [entry["model"], *lines].join("\n")
+      end
+      "#{blocks.join("\n\n")}\n"
+    end
+  end
+end

data/lib/deckard/status.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Deckard
+  # Writes progress and completion counts to standard error: a live counter
+  # while a terminal is attached, then a total line and per-type counts
+  # sorted by type name. Standard output is never touched - it belongs to
+  # the stream.
+  class Status
+    def self.progress(action, counts, stderr)
+      return unless stderr.tty?
+
+      stderr.write "\r#{action} #{counts.values.sum} objects"
+    end
+
+    def self.report(action, counts, stderr)
+      stderr.write "\r\e[K" if stderr.tty?
+      stderr.puts "#{action} #{counts.values.sum} total objects:"
+      return if counts.empty?
+
+      stderr.puts
+      name_width = counts.keys.map(&:length).max + 2
+      count_width = counts.values.max.to_s.length
+      counts.sort.each do |type, count|
+        stderr.puts "#{type.ljust(name_width)}#{count.to_s.rjust(count_width)}"
+      end
+    end
+  end
+end

data/lib/deckard/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Deckard
+  VERSION = "0.2.0"
+end

data/lib/deckard.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require_relative "deckard/version"
+require_relative "deckard/errors"
+require_relative "deckard/dumper"
+require_relative "deckard/loader"
+require_relative "deckard/model_config"
+require_relative "deckard/active_record/dump"
+require_relative "deckard/active_record/load"
+require_relative "deckard/active_record"
+
+require "active_support/lazy_load_hooks"
+ActiveSupport.on_load(:active_record) do
+  include Deckard::ActiveRecord
+end
+
+module Deckard
+  # Whether the booted application looks like production. The loader
+  # refuses to run there unless the operator passes --force.
+  def self.production_environment?
+    env = if defined?(::Rails) && ::Rails.respond_to?(:env)
+      ::Rails.env.to_s
+    else
+      ENV["RAILS_ENV"] || ENV["RACK_ENV"]
+    end
+    env == "production"
+  end
+
+  # Stream protocol frames. The header opens every stream; the end marker
+  # distinguishes a complete stream from one whose source died mid-dump.
+  STREAM_HEADER = [:deckard, 2].freeze
+  STREAM_END = [:deckard_end, 2].freeze
+end

metadata

@@ -0,0 +1,96 @@
+--- !ruby/object:Gem::Specification
+name: deckard
+version: !ruby/object:Gem::Version
+  version: 0.2.0
+platform: ruby
+authors:
+- Rob Sanheim
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+- !ruby/object:Gem::Dependency
+  name: optimist
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.2'
+description: Copy selected ActiveRecord records and their associations between PostgreSQL
+  databases, with destination-generated primary keys, remapped foreign keys, and transactional
+  loads.
+email:
+- rsanheim@gmail.com
+executables:
+- deckard
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- exe/deckard
+- lib/deckard.rb
+- lib/deckard/active_record.rb
+- lib/deckard/active_record/dump.rb
+- lib/deckard/active_record/load.rb
+- lib/deckard/cli.rb
+- lib/deckard/dumper.rb
+- lib/deckard/errors.rb
+- lib/deckard/loader.rb
+- lib/deckard/model_config.rb
+- lib/deckard/plan_report.rb
+- lib/deckard/status.rb
+- lib/deckard/version.rb
+homepage: https://github.com/rsanheim/deckard
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://github.com/rsanheim/deckard
+  source_code_uri: https://github.com/rsanheim/deckard/tree/main
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.21
+specification_version: 4
+summary: Stream ActiveRecord objects between Rails environments
+test_files: []