static_db 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6d592e282b1c5d72ea6a18964d5e9ac5e98fa8c42f9c417c11ea21454a8e6273
-  data.tar.gz: 136cb048f205ffb2bb54334ba3f95bc099fbbb17712a2afe9aea1487b565250b
+  metadata.gz: fda821f04db0db10b8c6beea35eb177028dedfb6ba8e2f22fbbddd5164ea4d30
+  data.tar.gz: 31bab97a76c1f75499d93fcbbe4e0a448109069d47ef85ad7cddf7dbf0f25957
 SHA512:
-  metadata.gz: 54feb805dff8f4e80228178f3c6d1180fa5d8be3ff23297824d711dc58656834fe68fc54fb9e1fff38a9b78db4e56ed8bc732e45c6b8c0dc4b009a3fdb7ef5ca
-  data.tar.gz: 455ce380cc64e703dfd1058742eb26c6568f0497e7c307dbff16a7cb803904b410cca2e104fe39021876ceee75631fe7b2a5c50c0d898dfc25d864b80d307ea0
+  metadata.gz: 69fbeebc35245d0a9d47d6ab0fc3e646e4e20dd4a98a01a12b5e706d0ea4155ae17656f171288feb86663ced7c6bdc3fe32ce8adf396ce9f04116fa5ce051dc9
+  data.tar.gz: 7debc1202b841a20e1149fc6898deb55ae998bb7ac14b78c019c6e59c00bda91b81069cec0af2be26b366a8210273097f9f86ce2a96779e31f4ff56480188b0a

data/CHANGELOG.md

@@ -6,11 +6,23 @@ This project adheres to [Break Versioning](https://www.taoensso.com/break-versio
 
 ### Breaking
 
--
+- Loading no longer goes through Rails fixtures. It inserts the YAML values as they are, so values that Rails fixtures used to change now load unchanged.
+  - ERB is no longer evaluated, and `$LABEL` is no longer replaced with the record's label.
+  - A missing key loads as `NULL` instead of the column default.
+  - Ids, timestamps and enum values are no longer filled in or looked up. If you add records to the YAML by hand, give them every column they need, including the id.
+- Dumping works per table instead of per model. Every table gets dumped, including tables without a model, except Rails' own `schema_migrations` and `ar_internal_metadata`.
+  - Labels are the table name and all primary key values, e.g. `posts_1` instead of `Post_1`. With a composite primary key, values containing `_` or `"` are quoted. This changes every label once.
+  - Tables without a primary key, e.g. `has_and_belongs_to_many` join tables, are dumped as a list of rows instead of a mapping.
 
 ### Non-breaking
 
--
+- Dump columns in alphabetical order, after the primary key columns. The order no longer depends on when a migration added a column. This reorders the keys in your YAML files once.
+- Fix JSON columns getting encoded a second time when loading.
+- Fix tables with generated columns: the dump leaves those columns out, so loading no longer fails on them.
+- Fix single-table inheritance: all rows of a table end up in its file, instead of each class overwriting the file with its own rows.
+- Support primary keys not called `id`, and composite primary keys.
+- Validate each row once, with the validations of its own class. Rows whose single-table inheritance class does not exist are reported as invalid.
+- Fix records hidden by a `default_scope`: they are now validated before dumping, and a table whose records are all hidden is no longer left out of the dump.
 
 ## [0.2.1] - 2026-05-08
 

data/README.md

@@ -1,5 +1,5 @@
 [![Gem Version](https://badge.fury.io/rb/static_db.svg)](https://rubygems.org/gems/static_db)
-[![Build](https://github.com/dunkelziffer/static_db/workflows/Build/badge.svg)](https://github.com/dunkelziffer/static_db/actions)
+[![CI](https://github.com/dunkelziffer/static_db/actions/workflows/ci.yml/badge.svg)](https://github.com/dunkelziffer/static_db/actions/workflows/ci.yml)
 
 # static_db
 
@@ -55,10 +55,44 @@ The boot sequence hooks are inactive in all other cases, e.g.:
 - When you run migrations: `bin/rails db:migrate`
 - When you run any other rake tasks
 
+## Data format
+
+Every table with rows gets a YAML file named after it, except Rails' own `schema_migrations` and `ar_internal_metadata`. Records are ordered by primary key and labeled with the table name and their primary key values, e.g. `posts_1`. Their columns come in alphabetical order, after the primary key columns. `NULL` values are left out.
+
+> ⚠️ A table without a primary key, like the join table of a `has_and_belongs_to_many` association, has nothing to label its rows with. It gets dumped as a list of rows instead, ordered by all columns, which makes for less readable diffs. Prefer tables with a primary key, e.g. `has_many :through` with a join model.
+
+## Workflows
+
+Both workflows run the rake tasks by hand, so make sure no server is running: it would load or dump on its own.
+
+### Applying a migration
+
+The hooks don't run for `db:migrate`, so load the YAML files first and dump the migrated database afterwards:
+
+```sh
+bin/rails static:load
+# write the migration
+bin/rails db:migrate static:dump
+```
+
+### Upgrading static_db
+
+When the [CHANGELOG](CHANGELOG.md) lists a breaking change, the new version may read or write your YAML files differently. Load them with the version that wrote them, then dump them with the new one:
+
+```sh
+bin/rails static:load # still on the old version
+bundle update static_db
+bin/rails static:dump
+```
+
+Review the diff before you commit it.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at [https://github.com/dunkelziffer/static_db](https://github.com/dunkelziffer/static_db).
 
+Run the tests with `bundle exec rspec` and the linter with `bundle exec rubocop`. Each directory in `spec/cases/` is a small app with its own schema, models and golden YAML files. After an intended change to the dump format, regenerate the golden files with `UPDATE_GOLDEN=1 bundle exec rspec` and review the diff.
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/lib/static_db/dump.rb

@@ -1,37 +1,55 @@
 module StaticDb
   class Dump
-    attr_reader :models_to_be_saved
+    # A table's rows as the database returns them, ordered by primary key, or by all
+    # columns for a table without one.
+    Table = Data.define(:name, :primary_key, :columns, :rows)
 
     def initialize(static_db_path: StaticDb.path)
       @static_db_path = Pathname.new(static_db_path)
-      @models_to_be_saved = models
     end
 
     def perform
       exit 1 if $skip_static_db_dump
 
-      validate_records!
+      tables = ActiveRecord::Base.with_connection { |connection| read_tables(connection) }
+      validate_records!(tables)
       reset_data_directory!
-      dump_data!
+      dump_data!(tables)
     end
 
     private
 
-    def validate_records!
-      puts green("Validating records ...")
+    # Every table, except Rails' own and empty ones.
+    def read_tables(connection)
+      (connection.tables - internal_tables).sort.filter_map do |name|
+        primary_key = connection.primary_keys(name)
+        columns = dumped_columns(connection.columns(name), primary_key)
+        order = (primary_key.presence || columns.map(&:name)).map { |column| connection.quote_column_name(column) }
+        rows = connection.select_all("SELECT * FROM #{connection.quote_table_name(name)} ORDER BY #{order.join(", ")}").to_a
 
-      errors = []
+        Table.new(name:, primary_key:, columns:, rows:) if rows.any?
+      end
+    end
 
-      models_to_be_saved.each do |model|
-        model.find_each do |record|
-          next if record.valid?
+    def internal_tables
+      base = ActiveRecord::Base
+      [ base.schema_migrations_table_name, base.internal_metadata_table_name ].map do |name|
+        "#{base.table_name_prefix}#{name}#{base.table_name_suffix}"
+      end
+    end
 
-          debug_data = { model: model.name, id: record.id, errors: record.errors.full_messages }
-          debug_data[:slug] = record.slug if record.respond_to?(:slug)
+    # Generated columns are left out: the database computes them and refuses values for them.
+    # Primary key columns come first, in the key's order, then the others alphabetically, so
+    # the order does not depend on when a migration added a column.
+    def dumped_columns(columns, primary_key)
+      columns.reject(&:virtual?).sort_by { |column| [ primary_key.index(column.name) || primary_key.size, column.name ] }
+    end
 
-          errors << debug_data
-        end
-      end
+    def validate_records!(tables)
+      puts green("Validating records ...")
+
+      Rails.application.eager_load!
+      errors = tables.flat_map { |table| validation_errors(table) }
 
       if errors.any?
         puts red("Found #{errors.length} invalid records!")
@@ -52,48 +70,73 @@ module StaticDb
       end
     end
 
+    # Validates every row of the table once per model that owns the table, as an instance of
+    # the row's own class: for single-table inheritance, Rails picks the subclass from the
+    # type column. Scopes do not apply, so no row is left out.
+    def validation_errors(table)
+      table_models(table.name).flat_map do |model|
+        table.rows.filter_map do |row|
+          record = model.instantiate(row)
+          next if record.valid?
+
+          error = { model: record.class.name, id: record.id, errors: record.errors.full_messages }
+          error[:slug] = record.slug if record.respond_to?(:slug)
+          error
+        rescue ActiveRecord::SubclassNotFound
+          type = model.inheritance_column
+          { model: model.name, id: row.values_at(*table.primary_key).join(", "), errors: [ "Unknown #{type} #{row[type].inspect}" ] }
+        end
+      end
+    end
+
+    # Models whose parent class does not use the same table: the base class of a
+    # single-table inheritance hierarchy, or any other model of that table.
+    def table_models(table_name)
+      ActiveRecord::Base.descendants.select do |model|
+        next false if model.abstract_class? || model.table_name != table_name
+
+        parent = model.superclass
+        parent == ActiveRecord::Base || parent.abstract_class? || parent.table_name != table_name
+      end
+    end
+
     def reset_data_directory!
       FileUtils.rm_rf(@static_db_path)
       FileUtils.mkdir_p(@static_db_path)
     end
 
-    def dump_data!
+    def dump_data!(tables)
       puts green("Dumping data ...")
 
-      models_to_be_saved.each { |model| write_file!(path: file_path(model), data: data(model)) }
+      tables.each { |table| File.write(@static_db_path.join("#{table.name}.yml"), records(table).to_yaml) }
 
       puts green("Done!")
     end
 
-    def write_file!(path:, data:)
-      File.open(path, "w") { |file| file << data.to_yaml }
-    end
+    # A mapping from labels to records, or a list of records for a table without a primary
+    # key, whose rows may not even be unique.
+    def records(table)
+      records = table.rows.map { |row| record(row, table.columns) }
+      return records if table.primary_key.empty?
 
-    def file_path(model)
-      File.join(@static_db_path, "#{model.table_name}.yml")
-    end
+      labels = table.rows.map { |row| label(table, row) }
+      raise "#{table.name}: labels are not unique" unless labels.uniq.size == labels.size
 
-    def data(model)
-      model.unscoped.order(:id).to_h do |instance|
-        [ "#{model}_#{instance.id}", instance_data(instance, model.columns) ]
-      end
+      labels.zip(records).to_h
     end
 
-    def instance_data(instance, columns)
-      columns.filter_map do |column|
-        value = instance.read_attribute_before_type_cast(column.name)
-        next if value.nil?
-
-        [ column.name, value ]
-      end.to_h
+    def record(row, columns)
+      columns.filter_map { |column| [ column.name, row[column.name] ] unless row[column.name].nil? }.to_h
     end
 
-    def models
-      Rails.application.eager_load!
+    # The table name and the primary key values, e.g. "posts_1" or "translations_de_hello".
+    # With more than one key column, a value containing "_" or '"' is quoted, so that
+    # ("a_b", "c") and ("a", "b_c") get different labels.
+    def label(table, row)
+      values = row.values_at(*table.primary_key).map(&:to_s)
+      values = values.map { |value| value.match?(/[_"]/) ? value.inspect : value } if values.size > 1
 
-      ActiveRecord::Base.descendants.select do |model|
-         model.table_exists? && model.any? && model != ActiveRecord::SchemaMigration
-      end
+      [ table.name, *values ].join("_")
     end
 
     def green(text)

data/lib/static_db/load.rb

@@ -24,17 +24,56 @@ module StaticDb
       puts green("Done!")
     end
 
+    # Inserts the rows exactly as they were dumped, one table per file. Unlike Rails fixtures:
+    # no ERB, no $LABEL, no generated ids or timestamps, no enum or association lookups, no
+    # type casting, and a missing column is NULL instead of the column default.
     def load_data!
       puts green("Loading data ...")
 
-      base_names = Dir.glob(File.join(@static_db_path, "*.yml")).map do |file|
-        File.basename(file, ".*")
+      tables = Dir.glob(@static_db_path.join("*.yml")).sort.to_h do |file|
+        [ File.basename(file, ".yml"), read_rows(file) ]
+      end
+
+      ActiveRecord::Base.with_connection do |connection|
+        connection.transaction do
+          connection.disable_referential_integrity do
+            tables.each { |table, rows| insert(connection, table, rows) }
+          end
+        end
+
+        connection.check_all_foreign_keys_valid!
+        tables.each_key { |table| connection.reset_pk_sequence!(table) } if connection.respond_to?(:reset_pk_sequence!)
       end
-      ActiveRecord::FixtureSet.create_fixtures(@static_db_path, base_names)
 
       puts green("Done!")
     end
 
+    # A mapping from labels to records, or a list of records for a table without a primary key.
+    def read_rows(file)
+      records = YAML.safe_load_file(file, permitted_classes: [ Date, Time ], aliases: true) || []
+      records = records.values if records.is_a?(Hash)
+      raise "#{file} is neither a mapping of labels to records nor a list of records" unless records.is_a?(Array) && records.all?(Hash)
+
+      records
+    end
+
+    def insert(connection, table, rows)
+      columns = connection.columns(table).reject(&:virtual?)
+      unknown = rows.flat_map(&:keys).uniq - columns.map(&:name)
+      raise "Cannot load #{table}.#{unknown.join(", ")}: no such column, or a generated one" if unknown.any?
+
+      column_list = columns.map { |column| connection.quote_column_name(column.name) }.join(", ")
+      rows.each_slice(1000) do |slice|
+        values = slice.map { |row| "(#{columns.map { |column| quote(connection, column, row[column.name]) }.join(", ")})" }
+        connection.execute("INSERT INTO #{connection.quote_table_name(table)} (#{column_list}) VALUES #{values.join(", ")}")
+      end
+    end
+
+    def quote(connection, column, value)
+      value = ActiveModel::Type::Binary::Data.new(value) if column.type == :binary && value.is_a?(String)
+      connection.quote(value)
+    end
+
     def green(text)
       "\e[32m#{text}\e[0m"
     end

data/lib/static_db/version.rb

@@ -1,3 +1,3 @@
 module StaticDb # :nodoc:
-  VERSION = "0.2.1"
+  VERSION = "0.3.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: static_db
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - Klaus Weidinger