cold_storage 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 2d7a93fbe18555d8eafe6a0720722476a369eea45bf86d81e7c43490c8a33456
+  data.tar.gz: 33399d9c88a64d53b7fff48a7ab75881454b2e77248be4247ad7e488b9f5ce12
+SHA512:
+  metadata.gz: 7147ede6d8a60b043f093f0ce1ab8000828f2eae5583d9a02d7efb7dd7588c88bdbf04d87a88e82faaedc004bf8df93f7c4ef22010cc1f31d101c258fb2d148c
+  data.tar.gz: 4b5954d3ad66ef1c48e2ad3d6db848633076310399df4fd49049f50860ea918f03b7782765926e6181f622a043fa29751c127b21f49807784eb121d15cd5cf26

data/CHANGELOG.md

@@ -0,0 +1,35 @@
+# Changelog
+
+## 0.1.0
+
+* `archivable` macro: `after:`, `on:`, `deleted:`, `deleted_column:`,
+  `on_destroy:`, `every:`, `scope:`, `cascade:`, `batch_size:`,
+  `delete_after_archive:`, `delete_method:`.
+* `on_destroy: true` keeps hard-deleted rows by copying them to the archive in
+  a `before_destroy` hook, together with the policy's cascade and without
+  deleting anything itself; `config.on_destroy_error` decides whether an
+  unreachable archive blocks the delete.
+* Association scopes that the archive class cannot evaluate degrade to the
+  unfiltered relation instead of raising on read.
+* Unique indexes are mirrored as plain indexes; the primary key is the only
+  uniqueness the archive enforces.
+* `config.enabled` master switch, so an app can stand the gem down per
+  environment (the test one, typically).
+* `cascade:` accepts a nested tree (`cascade: { lines: [:taxes] }`) so a whole
+  object graph can be taken along, validated before anything moves.
+* Separate archive database, connected lazily through `ColdStorage::ArchiveRecord`.
+* Schema mirroring for the archivable models only, including PostgreSQL enum
+  types, arrays and indexes; automatic sync after `db:migrate`, and a
+  `cold_storage:schema:check` task for CI.
+* Batched, idempotent archiving with cascade support, dry runs and limits.
+* Reading: `Model.archived`, `archived_count`, `find_archived`,
+  `find_with_archived`, and archive models that mirror the source
+  associations (`has_many`, `has_one`, `belongs_to`, `:through`) so an archived
+  row can be read together with its archived children.
+* Restore back into the primary database, on its own or with relations:
+  `with: :all`, `with: [:lines]`, `with: { lines: [:taxes] }`, from
+  `Model.restore_archived` or from an archived record's `restore!`.
+* Run bookkeeping in the archive database, which is what makes `every:` work
+  across processes.
+* `ColdStorage::ArchiveAllJob` / `ArchiveModelJob` and the
+  `cold_storage:*` rake tasks.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Afshin Amini
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,399 @@
+# Cold Storage
+
+Declarative auto-archiving for ActiveRecord.
+
+Add one line to a model and its old (or soft-deleted) rows move, on a schedule,
+into a **separate archive database** whose schema is a mirror of the primary
+one — created for you, and kept up to date on every migration, for the
+archivable models only.
+
+```ruby
+class Payroll < ApplicationRecord
+  archivable after: 18.months, every: 1.month
+end
+
+class Document < ApplicationRecord
+  archivable deleted: true, after: 30.days      # soft-deleted 30+ days ago
+end
+
+class Receipt < ApplicationRecord
+  archivable on_destroy: true                   # keep hard-deleted rows
+end
+
+class Invoice < ApplicationRecord
+  archivable after: 2.years, cascade: [:invoice_lines]
+end
+```
+
+## Installation
+
+```ruby
+# Gemfile
+gem 'cold_storage', git: 'git@github.com:afshmini/cold_storage.git'
+# or, while working on it:
+gem 'cold_storage', path: '../cold_storage'
+```
+
+```bash
+rails generate cold_storage:install
+```
+
+### 1. Add the archive database
+
+`config/database.yml`, in every environment that should archive:
+
+```yaml
+development:
+  primary:
+    <<: *default
+  archive:
+    <<: *default
+    database: <%= ENV.fetch('ARCHIVE_POSTGRES_DB') %>
+    database_tasks: false
+```
+
+`database_tasks: false` matters: it stops `rails db:migrate` from running your
+application migrations against the archive database. ColdStorage mirrors the
+schema itself, and only for the tables it needs.
+
+### 2. Create it and mirror the schema
+
+```bash
+rails cold_storage:db:create     # CREATE DATABASE
+rails cold_storage:schema:sync   # copy the archivable models' tables
+rails cold_storage:status
+```
+
+### 3. Schedule it
+
+The bundled job walks every archivable model and lets each one decide whether
+its `every:` window has elapsed, so a daily schedule is enough:
+
+```yaml
+# config/recurring.yml (solid_queue)
+cold_storage:
+  schedule: "0 2 * * *"
+  class: "ColdStorage::ArchiveAllJob"
+```
+
+Or from cron/rake: `rails cold_storage:archive_all`.
+
+## The `archivable` options
+
+| Option | Meaning | Default |
+| --- | --- | --- |
+| `after:` (alias `older_than:`) | archive rows older than this duration | — |
+| `on:` | column the age is measured on | `:created_at`, or the deleted column when `deleted: true` |
+| `deleted:` | archive soft-deleted rows only | `false` |
+| `deleted_column:` | where the soft-delete timestamp lives | `config.deleted_column` (`:deleted_at`) |
+| `on_destroy:` (alias `hard_delete:`) | copy a row to the archive whenever it is really destroyed | `false` |
+| `every:` | minimum time between two runs of this model | run on every pass |
+| `scope:` | symbol, proc or relation narrowing the selection | — |
+| `cascade:` | `has_many` / `has_one` names archived with the parent, nestable | `[]` |
+| `batch_size:` | rows per round trip | `config.batch_size` (1000) |
+| `delete_after_archive:` | remove the source rows once copied | `true` |
+| `delete_method:` | `:delete_all` or `:destroy_all` | `:delete_all` |
+
+At least one of `after:`, `deleted:`, `scope:` or `on_destroy:` is required —
+without a criterion the whole table would be archivable, which is never what
+you meant.
+
+```ruby
+archivable after: 12.months                                # by age
+archivable after: 90.days, on: :closed_at                  # by another column
+archivable deleted: true                                   # soft-deleted rows
+archivable deleted: true, after: 30.days                   # ... after a grace period
+archivable on_destroy: true                                # hard deletes
+archivable after: 5.years, on_destroy: true                # both
+archivable after: 5.years, every: 1.month                  # ... at most monthly
+archivable after: 1.year, scope: -> { where(exported: true) }
+archivable after: 2.years, cascade: [:items], batch_size: 5_000
+archivable after: 2.years, cascade: [{ items: [:taxes, :notes] }, :comments]
+```
+
+`cascade:` takes a name, an array, or a nested hash, as deep as the graph goes.
+Children are archived and deleted before their parents, so foreign keys hold at
+every step, and anything the cascade does not name is left behind — which for a
+child with a `NOT NULL` foreign key means the parent's delete fails, loudly.
+Association scopes are ignored on the way down: every row pointing at the
+parent is taken, never a subset, so nothing is orphaned.
+
+The selection always starts from `unscoped`, so a soft-delete `default_scope`
+cannot hide the very rows you asked to archive.
+
+## Soft delete, hard delete, both
+
+Three different things can make a row disappear, and each has its own switch:
+
+* **it got old** — `after:` sweeps it on a schedule and then deletes it from
+  the primary database;
+* **it was soft-deleted** — `deleted: true` picks up rows whose `deleted_at`
+  is set (optionally after a grace period with `after:`);
+* **it is being hard-deleted right now** — `on_destroy: true` copies the row to
+  the archive in a `before_destroy` hook, so `destroy`, `destroy!` and
+  `destroy_all` keep a copy instead of losing one.
+
+```ruby
+class Receipt < ApplicationRecord
+  archivable on_destroy: true                  # nothing swept, deletes kept
+end
+
+class Payroll < ApplicationRecord
+  archivable after: 18.months, on_destroy: true # aged out *and* deleted early
+end
+```
+
+`on_destroy:` on its own means "react to deletes", not "archive this table":
+a scheduled run reports the model as skipped (`:destroy_only`) and
+`archivable_records` is empty, so nothing is swept behind your back.
+
+Worth knowing:
+
+* `delete`, `delete_all` and database-level `ON DELETE CASCADE` do not run
+  callbacks, so they cannot be captured. Use `destroy`/`destroy_all`, or
+  `dependent: :destroy` on the parent, for rows you must keep.
+* The parent's `cascade:` comes along, because `dependent: :destroy` children
+  are about to go too. Nothing is deleted by the hook itself: the destroy that
+  triggered it is what removes the rows, so `dependent:` keeps deciding what
+  happens to the children (and a rolled-back destroy loses nothing).
+* If the archive database cannot be reached, the destroy fails
+  (`config.on_destroy_error = :raise`, the default): losing the row is worse
+  than failing the delete. Set it to `:log` to let deletes through and only
+  record the problem.
+* The copy is written before the surrounding transaction commits, so a
+  rolled-back destroy can leave a copy in the archive. It is keyed by primary
+  key, so the row is still identifiable and a later real archive overwrites it.
+* Each destroyed record is one round trip to the archive database; mass
+  cleanups are better served by the scheduled sweep.
+
+## What you get on the model
+
+```ruby
+Payroll.archivable_records      # relation of rows eligible right now
+Payroll.archivable_count
+Payroll.archive_now!            # archive immediately, ignoring every:
+payroll.archive!                # a single record (with its cascade)
+payroll.archived?
+
+ColdStorage.archive(Payroll)                 # honours every:
+ColdStorage.archive(Payroll, dry_run: true)  # report, move nothing
+ColdStorage.archive_all
+```
+
+## Reading archived data
+
+```ruby
+Payroll.archived                       # relation on the archive database
+Payroll.archived.where(year: 2019).order(:id).pluck(:total)
+Payroll.archived_count
+Payroll.find_archived(42)              # RecordNotFound if it is not archived
+Payroll.find_with_archived(42)         # live record, or the archived one
+```
+
+`archived` is an ordinary relation on a class connected to the archive
+database, so scopes, `where`, `pluck`, `find_each` and `includes` all work.
+**The model's associations are mirrored onto it**, pointing at the archive
+database, so an archived row can be read together with its archived children:
+
+```ruby
+invoice = Invoice.archived.find(42)
+invoice.invoice_lines                  # archived lines, from the archive DB
+invoice.taxes                          # has_many :through works too
+Invoice.archived.includes(:invoice_lines).find_each { |i| ... }
+
+line = ColdStorage.archived_model(InvoiceLine).find(7)
+line.invoice                           # ... and back up
+line.source_model                      # => InvoiceLine
+```
+
+Association scopes carry over when the archive class can evaluate them
+(`-> { where(active: true) }`). One that reaches for something only the source
+model has (`-> { sorted('start_time', 'asc') }`, a scope on the child) falls
+back to the unfiltered relation instead of raising, so a read never dies on a
+scope. Turn on debug logging to see which ones were ignored.
+
+Mirrored: `has_many`, `has_one`, `belongs_to` and `:through`. Not mirrored:
+polymorphic `belongs_to`, and `:through` with a `source_type:` — both would
+have to resolve a class name back into the primary database, and reading would
+silently cross into it. `dependent:`, counter caches and `touch` are dropped on
+purpose: the archive is a reading surface and must not cascade anything.
+
+Joins between the two databases are not possible — they are separate
+connections.
+
+## Recovering
+
+```ruby
+ColdStorage.restore(Payroll, [1, 2, 3])
+Payroll.restore_archived([1, 2, 3])                       # same thing
+Payroll.restore_archived(Payroll.archived.where(year: 2019))
+Invoice.archived.find(42).restore!                        # from the row itself
+
+# ... with its relations
+Invoice.restore_archived([42], with: :all)                # every archived child
+Invoice.restore_archived([42], with: [:invoice_lines])    # named relations
+Invoice.restore_archived([42], with: { invoice_lines: [:taxes] })  # nested
+Invoice.archived.find(42).restore!(with: :all)
+
+# keep the archived copy instead of moving the rows
+Invoice.restore_archived([42], with: :all, delete_from_archive: false)
+```
+
+Rows are written to the primary database parent-first, so foreign keys hold at
+every step, and the archived copies are removed as they land. The return value
+is the number of restored rows, children included.
+
+`with: :all` walks every `has_many`/`has_one` that has an archive table,
+recursively, cutting cycles as it goes. Named associations are checked before
+anything is written, so a typo cannot leave half a graph behind. A `belongs_to`
+is refused: restore the owner first, then its children — restoring a child
+whose parent is still archived would fail on the foreign key.
+
+```
+rails cold_storage:restore[Invoice,"42 43"] WITH=all
+rails cold_storage:restore[Invoice,"42"]    WITH=invoice_lines,taxes
+```
+
+## Rake tasks
+
+```
+rails cold_storage:status                 # models, rules, pending/archived counts, last run
+rails cold_storage:db:create              # create the archive database
+rails cold_storage:schema:sync            # create/update the mirrored tables
+rails cold_storage:schema:plan            # what sync would change
+rails cold_storage:schema:check           # exit 1 on drift (for CI)
+rails cold_storage:archive[Payroll]       # one model, now
+rails cold_storage:archive_all            # every model whose every: elapsed
+rails cold_storage:restore[Payroll,"1 2"] # move rows back (WITH=all)
+```
+
+`DRY_RUN=true`, `LIMIT=1000` and `FORCE=true` are honoured by the archive tasks.
+
+## Schema mirroring
+
+`cold_storage:schema:sync` walks the archivable models (plus the tables they
+cascade into) and, in the archive database:
+
+* creates missing tables, column by column, with the same SQL types —
+  including PostgreSQL enum types, arrays and `jsonb`;
+* keeps the primary key, so re-archiving a row updates it instead of
+  duplicating it;
+* mirrors indexes, **dropping their uniqueness** — an archive accumulates
+  history, and a natural key that is unique in the primary database at any one
+  moment is not unique across everything that table ever held;
+* adds an `archived_at` column (configurable, `nil` disables it);
+* adds columns that later migrations introduced;
+* records the migration version it synced against.
+
+and deliberately does **not**:
+
+* copy foreign keys — an archive holds partial object graphs;
+* copy `NOT NULL` or defaults — a schema that gets stricter later must not
+  reject rows that were archived before that (`mirror_null_constraints`,
+  `mirror_defaults` if you disagree);
+* drop columns the primary database dropped — archived rows keep the columns
+  they were archived with (`drop_removed_columns` if you disagree).
+
+It runs automatically after `db:migrate`, `db:rollback` and `db:schema:load`
+(`config.sync_schema_after_migrate`). In CI, `cold_storage:schema:check`
+fails the build when a migration changed an archivable table and the archive
+database was not brought along.
+
+Column type changes are reported, not applied, unless you ask:
+`config.on_type_mismatch = :warn | :raise | :change | :ignore`.
+
+## How a row moves
+
+Two databases cannot share a transaction, so each batch is:
+
+1. read from the primary database, cast by the **database** column types (model
+   level serializers, enums and default scopes are bypassed on both sides, so
+   what lands in the archive is byte-for-byte what was in the source column);
+2. `upsert`ed into the archive database, keyed on the primary key — a retry can
+   never duplicate a row;
+3. deleted from the primary database.
+
+A crash between 2 and 3 leaves the row in both databases; the next run upserts
+it again and deletes it. Archiving is at-least-once, and never loses a row.
+
+With `cascade:`, children are archived and deleted **before** their parent, so
+foreign keys in the primary database hold at every step.
+
+`delete_method: :delete_all` (the default) skips callbacks by design: `after_destroy`
+hooks that notify, bill or cascade should not fire because a row aged out. Use
+`:destroy_all` when you do want them.
+
+## Scheduling and `every:`
+
+Every successful run is recorded in `cold_storage_runs` **in the archive
+database**, and `every:` is checked against it. This means the cadence survives
+restarts and deploys, and two workers running `archive_all` on the same day do
+not archive twice.
+
+`ColdStorage::ArchiveAllJob` enqueues one `ArchiveModelJob` per model, so a
+big table cannot starve the others.
+
+## Configuration
+
+```ruby
+ColdStorage.configure do |config|
+  config.enabled                   = !Rails.env.test?
+  config.archive_database          = :archive
+  config.batch_size                = 1_000
+  config.timestamp_column          = :created_at
+  config.deleted_column            = :deleted_at
+  config.archived_at_column        = :archived_at   # nil to disable
+  config.delete_after_archive      = true
+  config.delete_method             = :delete_all
+  config.mirror_indexes            = true
+  config.mirror_null_constraints   = false
+  config.mirror_defaults           = false
+  config.drop_removed_columns      = false
+  config.on_type_mismatch          = :warn
+  config.on_destroy_error          = :raise         # or :log
+  config.sync_schema_after_migrate = true
+  config.throttle                  = 0              # seconds between batches
+  config.job_queue                 = :default
+  config.job_parent_class          = 'ApplicationJob'
+  config.dry_run                   = false
+end
+```
+
+`config.enabled = false` stands everything down: scheduled runs report
+themselves as skipped and the `on_destroy` hook does nothing. That is usually
+what you want in the test environment, where there is no archive database and
+specs destroy records all the time. Schema tasks keep working either way, so
+`cold_storage:schema:check` still guards CI.
+
+## Requirements and limits
+
+* Rails 7.1+, Ruby 3.1+. Developed and tested against PostgreSQL; the enum
+  mirroring is PostgreSQL specific and simply does nothing elsewhere.
+* Models need a single-column primary key.
+* Custom PostgreSQL types other than enums (domains, composites) are not
+  mirrored.
+* Views, functions and triggers are not mirrored.
+* `cascade:` supports `has_many`/`has_one`, including polymorphic children;
+  `:through` associations are rejected.
+
+## Tests
+
+The suite runs against two real PostgreSQL databases, because enums, arrays and
+`jsonb` are exactly what a schema mirror gets wrong.
+
+```bash
+bundle install
+bundle exec rspec
+```
+
+It reads `DB_HOST`, `DB_PORT`, `POSTGRES_USER` and `POSTGRES_PASSWORD`, and
+creates `cold_storage_source_test` / `cold_storage_archive_test`.
+
+To run it against the PostgreSQL of an app that already has a container, copy
+the gem in and borrow that app's bundle:
+
+```bash
+docker cp . <app-container>:/tmp/ra
+docker exec -w /tmp/ra <app-container> \
+  bash -lc 'BUNDLE_GEMFILE=/path/to/app/Gemfile bundle exec rspec'
+```

data/lib/cold_storage/archivable.rb

@@ -0,0 +1,144 @@
+# frozen_string_literal: true
+
+module ColdStorage
+  # Mixed into a model the first time it declares `archivable`.
+  module Archivable
+    extend ActiveSupport::Concern
+
+    included do
+      class_attribute :archiving_policy, instance_accessor: false, default: nil
+      class_attribute :archiving_destroy_hook, instance_accessor: false, default: false
+    end
+
+    class_methods do
+      def archivable?
+        archiving_policy.present?
+      end
+
+      # Copies a row to the archive just before it is really deleted.
+      # Installed by `archivable on_destroy: true`.
+      def install_archive_on_destroy!
+        return if archiving_destroy_hook
+
+        self.archiving_destroy_hook = true
+        # prepend: so the row is captured before dependent: callbacks start
+        # taking its children apart.
+        before_destroy(:archive_before_destroy, prepend: true)
+      end
+
+      # Rows that are eligible for archiving right now.
+      # @return [ActiveRecord::Relation]
+      def archivable_records
+        archiving_policy!.relation(self)
+      end
+
+      # How many rows are waiting to be archived.
+      def archivable_count
+        archivable_records.count
+      end
+
+      # Handle on the archived rows, living in the archive database. An
+      # ordinary relation: scopes, where, pluck, find_each all work, and the
+      # model's associations are mirrored, so archived children come along.
+      #
+      #   Payroll.archived.where(company_id: 1).count
+      #   Invoice.archived.find(42).invoice_lines
+      #
+      # @return [ActiveRecord::Relation]
+      def archived
+        archive_model.all
+      end
+
+      # How many rows of this model sit in the archive.
+      def archived_count
+        archived.count
+      end
+
+      # @return [ActiveRecord::Base] the archived row
+      # @raise [ActiveRecord::RecordNotFound]
+      def find_archived(*ids)
+        archive_model.find(*ids)
+      end
+
+      # Looks in the primary database first, then in the archive.
+      #
+      # @return [ActiveRecord::Base] a live record, or an archived one
+      # @raise [ActiveRecord::RecordNotFound] when it is in neither
+      def find_with_archived(id)
+        unscoped.find_by(primary_key => id) || find_archived(id)
+      end
+
+      # Moves archived rows back into this table.
+      #
+      #   Payroll.restore_archived([1, 2, 3])
+      #   Invoice.restore_archived(Invoice.archived.where(year: 2019), with: :all)
+      #
+      # @return [Integer] number of restored rows, children included
+      def restore_archived(ids, **options)
+        ColdStorage.restore(self, ids, **options)
+      end
+
+      # The ActiveRecord class mapped onto this table in the archive database.
+      # @return [Class]
+      def archive_model
+        ArchiveModel.for(self)
+      end
+
+      # Archives this model now, ignoring the `every:` window by default.
+      # @return [ColdStorage::Archiver::Result]
+      def archive_now!(**options)
+        Archiver.new(self, **{ force: true }.merge(options)).call
+      end
+
+      # @return [ColdStorage::Policy]
+      def archiving_policy!
+        archiving_policy || raise(NotArchivableError, "#{name} is not archivable")
+      end
+    end
+
+    # Archives this single record (and its cascaded children).
+    # @return [ColdStorage::Archiver::Result]
+    def archive!
+      relation = self.class.unscoped.where(self.class.primary_key => id)
+      Archiver.new(self.class, relation: relation, force: true).call
+    end
+
+    # Is there a row with this id in the archive database?
+    def archived?
+      self.class.archive_model.where(self.class.primary_key => id).exists?
+    end
+
+    private
+
+    # Keeps a copy of a hard-deleted row. The row is still readable here, so it
+    # is read (and archived) exactly like the scheduled run would.
+    #
+    # The policy's cascade comes along, because `dependent: :destroy` children
+    # are about to go too. Nothing is deleted here: the destroy that triggered
+    # this is what removes the rows, so `dependent:` keeps deciding what
+    # happens to the children.
+    #
+    # Raises by default: losing the row is worse than failing the delete. Set
+    # `config.on_destroy_error = :log` to let deletes through instead.
+    def archive_before_destroy
+      return unless ColdStorage.config.enabled
+      return unless self.class.archiving_policy&.on_destroy?
+      return if new_record? || id.nil?
+
+      Archiver.new(
+        self.class,
+        relation: self.class.unscoped.where(self.class.primary_key => id),
+        force: true,
+        track: false,
+        cascade: true,
+        delete_after_archive: false
+      ).call
+    rescue StandardError => e
+      raise if ColdStorage.config.on_destroy_error == :raise
+
+      ColdStorage.logger.error do
+        "#{Logging::PREFIX} could not archive #{self.class}##{id} before destroy: #{e.class}: #{e.message}"
+      end
+    end
+  end
+end