historiographer 4.0.0 → 4.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f16398b9f6b0a3d58501a2115eabca0e0f93a938d04eeb4c21d3713f9b70079a
-  data.tar.gz: a460ef65372e236b001225b1028837eb7ba107e8d45d958c39d003be0cd64db4
+  metadata.gz: 740453114d73fc300d55d887a57f8bbf447042bb5d65da886125ece34982bf33
+  data.tar.gz: 201cfd99a05408c4f4387d4c7244d87c985ce82c7aaab24db8114c7760831f31
 SHA512:
-  metadata.gz: 3b545edcfe0076f0e83f5478347a9298fce77118d7274ad39ce8304844637b7ee15980e66c02dff36321eda4e36b0defdb10b6c33cc7486fba47e780043e5f42
-  data.tar.gz: f828cf26cc357fb0b3e2750bfcc1f62dc628c1579f1fa3356062f34314545783bb6ecb80366e4ac9d6988eae8374168afeb0d29c166daa9dbd7c860074494de4
+  metadata.gz: 93d61f00c711ef9ef058e1e3eef9444755ea24756d5bb139cd2f657396061f80d9f405ed6eef4d86164fad8b10abd7e9103d0776556a2d1d344b379dab9d6076
+  data.tar.gz: acfe88c0b2fad42d24484b62fcd3cae84149eca55d0aea1d1f6771a5c6e785508679a5430f533d427c6aa460a8a5f355023f50ad42614282ae5d7b88591aaf19

data/README.md

@@ -8,21 +8,11 @@ Historiographer fixes this problem in a better way than existing auditing gems.
 
 The Audited gem has some serious flaws.
 
-🤚Hands up if your `versions` table has gotten too big to query 🤚
+1. The `versions` table quickly grows too large to query
 
-🤚Hands up if your `versions` table doesn't have the indexes you need 🤚
+2. It doesn't provide the indexes you need from your primary tables
 
-🤚Hands up if you've ever iterated over `versions` records in Ruby to recreate a snapshot of what data looked like at a point in time. 🤚
-
-Why does this happen?
-
-First, `audited` only tracks a record of what changed, so there's no way to "go back in time" and see what the data looked like back when a problem occurred without replaying every single audit.
-
-Second, it tracks changes as JSON. While some data stores have JSON querying semantics, not all do, making it very hard to ask complex questions of your historical data -- that's the whole reason you're keeping it around.
-
-Third, it doesn't maintain indexes on your data. If you maintain an index on the primary table, wouldn't you also want to look up historical records using the same columns? Historical data is MUCH larger than "latest snapshot" data, so, duh, of course you do.
-
-Finally, Audited creates just one table for all audits. Historical data is big. It's not unusual for an audited gem table to get into the many millions of rows, and need to be constantly partitioned to maintain any kind of efficiency.
+3. It doesn't provdie out-of-the-box snapshots
 
 ## How does Historiographer solve these problems?
 
@@ -30,44 +20,116 @@ Historiographer introduces the concept of _history tables:_ append-only tables t
 
 If you have a `posts` table:
 
-| id | title |
-| :----------- | :----------- |
-| 1      | My Great Post       |
-| 2 | My Second Post |
+| id  | title          |
+| :-- | :------------- |
+| 1   | My Great Post  |
+| 2   | My Second Post |
 
 You'll also have a `post_histories_table`:
 
-| id | post_id | title | history_started_at | history_ended_at | history_user_id |
-| :----------- | :----------- | :----------- | :----------- | :----------- | :----------- |
-| 1      | 1 | My Great Post | '2019-11-08' | NULL | 1 |
-| 2 | 2| My Second Post | '2019-11-08' | NULL | 1 |
+| id  | post_id | title          | history_started_at | history_ended_at | history_user_id |
+| :-- | :------ | :------------- | :----------------- | :--------------- | :-------------- |
+| 1   | 1       | My Great Post  | '2019-11-08'       | NULL             | 1               |
+| 2   | 2       | My Second Post | '2019-11-08'       | NULL             | 1               |
 
 If you change the title of the 1st post:
 
-```Post.find(1).update(title: "Title With Better SEO", history_user_id: current_user.id)```
+`Post.find(1).update(title: "Title With Better SEO", history_user_id: current_user.id)`
 
 You'll expect your `posts` table to be updated directly:
 
-| id | title |
-| :----------- | :----------- |
-| 1      | Title With Better SEO |
-| 2 | My Second Post |
+| id  | title                 |
+| :-- | :-------------------- |
+| 1   | Title With Better SEO |
+| 2   | My Second Post        |
 
 But also, your `histories` table will be updated:
 
-| id | post_id | title | history_started_at | history_ended_at | history_user_id |
-| :----------- | :----------- | :----------- | :----------- | :----------- | :----------- |
-| 1      | 1 | My Great Post | '2019-11-08' | '2019-11-09' | 1 |
-| 2 | 2| My Second Post | '2019-11-08' | NULL | 1 |
-| 1      | 1 | Title With Better SEO | '2019-11-09' | NULL | 1 |
+| id  | post_id | title                 | history_started_at | history_ended_at | history_user_id |
+| :-- | :------ | :-------------------- | :----------------- | :--------------- | :-------------- |
+| 1   | 1       | My Great Post         | '2019-11-08'       | '2019-11-09'     | 1               |
+| 2   | 2       | My Second Post        | '2019-11-08'       | NULL             | 1               |
+| 1   | 1       | Title With Better SEO | '2019-11-09'       | NULL             | 1               |
 
 A few things have happened here:
 
 1. The primary table (`posts`) is updated directly
 2. The existing history for `post_id=1` is timestamped when its `history_ended_at`, so that we can see when the post had the title "My Great Post"
-3. A new history record is appended to the table containing a complete snapshot of the record, and a `NULL` `history_ended_at`. That's because this is the current history. 
+3. A new history record is appended to the table containing a complete snapshot of the record, and a `NULL` `history_ended_at`. That's because this is the current history.
 4. A record of _who_ made the change is saved (`history_user_id`). You can join to your users table to see more data.
 
+## Snapshots
+
+Snapshots are particularly useful for two key use cases:
+
+### 1. Time Travel & Auditing
+
+When you need to see exactly what your data looked like at a specific point in time - not just individual records, but entire object graphs with all their associations. This is invaluable for:
+
+- Debugging production issues ("What did the entire order look like when this happened?")
+- Compliance requirements ("Show me the exact state of this patient's record on January 1st")
+- Auditing complex workflows ("What was the state of this loan application when it was approved?")
+
+### 2. Machine Learning & Analytics
+
+When you need immutable snapshots of data for:
+
+- Training data versioning
+- Feature engineering
+- Model validation
+- A/B test analysis
+- Ensuring reproducibility of results
+
+### Taking Snapshots
+
+You can take a snapshot of a record and all its associated records:
+
+```ruby
+post = Post.find(1)
+post.snapshot(history_user_id: current_user.id)
+```
+
+This will:
+
+1. Create a history record for the post
+2. Create history records for all associated records (comments, author, etc.)
+3. Link these history records together with a shared `snapshot_id`
+
+You can retrieve the latest snapshot using:
+
+```ruby
+post = Post.find(1)
+snapshot = post.latest_snapshot
+
+# Access associated records from the snapshot
+snapshot.comments # Returns CommentHistory records
+snapshot.author   # Returns AuthorHistory record
+```
+
+Snapshots are immutable - you cannot modify history records that are part of a snapshot. This guarantees that your historical data remains unchanged, which is crucial for both auditing and machine learning applications.
+
+### Snapshot-Only Mode
+
+If you want to only track snapshots and not record every individual change, you can configure Historiographer to operate in snapshot-only mode:
+
+```ruby
+Historiographer::Configuration.mode = :snapshot_only
+```
+
+In this mode:
+
+- Regular updates/changes will not create history records
+- Only explicit calls to `snapshot` will create history records
+- Each snapshot still captures the complete state of the record and its associations
+
+This can be useful when:
+
+- You only care about specific points in time rather than every change
+- You want to reduce the number of history records created
+- You need to capture the state of complex object graphs at specific moments
+- You're versioning training data for machine learning models
+- You need to maintain immutable audit trails at specific checkpoints
+
 # Getting Started
 
 Whenever you include the `Historiographer` gem in your ActiveRecord model, it allows you to insert, update, or delete data as you normally would.
@@ -78,18 +140,33 @@ class Post < ActiveRecord::Base
 end
 ```
 
-By default, Historiographer will require all SQL-backed methods to provide a `history_user_id` to track who made the changes.
+### History Modes
+
+Historiographer supports two modes of operation:
+
+1. **:histories mode** (default) - Records history for every change to a record
+2. **:snapshot_only mode** - Only records history when explicitly taking snapshots
+
+You can configure the mode globally:
 
 ```ruby
-Post.create(title: "My Post", history_user_id: current_user.id) # => OK
-Post.create(title: "My Post") # => Error!
+# In an initializer
+Historiographer::Configuration.mode = :histories  # Default mode
+# or
+Historiographer::Configuration.mode = :snapshot_only
 ```
 
-Many existing models will be better off using `Historiographer::Safe` when getting started, which will not raise an error, but will alert you of locations where your app is missing `history_user_id`. 
+Or per model using `historiographer_mode`:
 
 ```ruby
 class Post < ActiveRecord::Base
-  include Historiographer::Safe
+  include Historiographer
+  historiographer_mode :snapshot_only  # Only record history when .snapshot is called
+end
+
+class Comment < ActiveRecord::Base
+  include Historiographer
+  historiographer_mode :histories  # Record history for every change (default)
 end
 ```
 
@@ -210,12 +287,11 @@ end
 == Mysql Install
 
 For contributors on OSX, you may have difficulty installing mysql:
- 
+
 ```
 gem install mysql2 -v '0.4.10' --source 'https://rubygems.org/' -- --with-ldflags=-L/usr/local/opt/openssl/lib --with-cppflags=-I/usr/local/opt/openssl/include
 ```
 
-
 == Copyright
 
 Copyright (c) 2016-2020 brettshollenberger. See LICENSE.txt for

data/lib/historiographer/configuration.rb

@@ -0,0 +1,36 @@
+require "singleton"
+
+module Historiographer
+  class Configuration
+    include Singleton
+
+    OPTS = {
+      mode: {
+        default: :histories
+      }
+    }
+    OPTS.each do |key, options|
+      attr_accessor key
+    end
+
+    class << self
+      def configure
+        yield instance
+      end
+
+      OPTS.each do |key, options|
+        define_method "#{key}=" do |value|
+          instance.send("#{key}=", value)
+        end
+
+        define_method key do
+          instance.send(key) || options.dig(:default)
+        end
+      end
+    end
+
+    def initialize
+      @mode = :histories
+    end
+  end
+end

data/lib/historiographer/history.rb

@@ -139,7 +139,7 @@ module Historiographer
       # If the record was not already persisted, proceed as normal.
       #
       def save(*args)
-        if persisted? && (changes.keys - %w(history_ended_at)).any?
+        if persisted? && (changes.keys - %w(history_ended_at snapshot_id)).any?
           false
         else
           super
@@ -147,12 +147,19 @@ module Historiographer
       end
 
       def save!(*args)
-        if persisted? && (changes.keys - %w(history_ended_at)).any?
+        if persisted? && (changes.keys - %w(history_ended_at snapshot_id)).any?
           false
         else
           super
         end
       end
+
+      # Returns the most recent snapshot for each snapshot_id
+      # Orders by history_started_at and id to handle cases where multiple records
+      # have the same history_started_at timestamp
+      scope :latest_snapshot, -> {
+        where.not(snapshot_id: nil).order('id DESC').limit(1)&.first
+      }
     end
 
     class_methods do

data/lib/historiographer/history_migration.rb

@@ -29,22 +29,24 @@ module Historiographer
       original_columns = klass.columns.reject { |c| c.name == "id" || except.include?(c.name) || (only.any? && only.exclude?(c.name)) || no_business_columns }
 
       integer foreign_key.to_sym, null: false
+      valid_keys = [:limit, :precision, :scale, :default, :null, :collation, :comment, 
+                    :primary_key, :if_exists, :if_not_exists, :array, :using, 
+                    :cast_as, :as, :type, :enum_type, :stored]
 
       original_columns.each do |column|
-        opts = {}
-        opts.merge!(column.as_json.clone)
+        opts = column.as_json.symbolize_keys.slice(*valid_keys) # Only keep valid keys
 
         if RUBY_VERSION.to_i >= 3
-          puts "Hello"
-          send(column.type, column.name, **opts.symbolize_keys!)
+          send(column.type, column.name, **opts)
         else
-          send(column.type, column.name, opts.symbolize_keys!)
+          send(column.type, column.name, opts)
         end
       end
 
       datetime :history_started_at, null: false
       datetime :history_ended_at
       integer :history_user_id
+      string :snapshot_id
 
       indices_sql = %q(
         SELECT 
@@ -75,7 +77,8 @@ module Historiographer
         foreign_key,
         :history_started_at,
         :history_ended_at,
-        :history_user_id
+        :history_user_id,
+        :snapshot_id
       ])
 
       indexes.each do |index_definition|

data/lib/historiographer/relation.rb

@@ -11,7 +11,7 @@ module Historiographer
     end
 
     def update_all(updates, histories=true)
-      unless histories
+      if !histories || self.model.is_history_class?
         super(updates)
       else
         updates.symbolize_keys!

data/lib/historiographer/version.rb

@@ -0,0 +1,3 @@
+module Historiographer
+  VERSION = "4.1.0"
+end

data/lib/historiographer.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'active_support/all'
+require 'securerandom'
 require_relative './historiographer/history'
 require_relative './historiographer/postgres_migration'
 require_relative './historiographer/safe'
@@ -83,12 +84,21 @@ module Historiographer
     after_save :record_history, if: :should_record_history?
     validate :validate_history_user_id_present, if: :should_validate_history_user_id_present?
 
+    # Add scope to fetch latest histories
+    scope :latest_snapshot, -> {
+      history_class.latest_snapshot
+    }
+
+    def should_alert_history_user_id_present?
+      !snapshot_mode? && !is_history_class? && Thread.current[:skip_history_user_id_validation] != true
+    end
+
     def should_validate_history_user_id_present?
-      true
+      !snapshot_mode? && !is_history_class? && Thread.current[:skip_history_user_id_validation] != true
     end
 
     def validate_history_user_id_present
-      if @no_history.nil? && (!history_user_id.present? || !history_user_id.is_a?(Integer))
+      if should_validate_history_user_id_present? && (@no_history.nil? && (!history_user_id.present? || !history_user_id.is_a?(Integer)))
         errors.add(:history_user_id, 'must be an integer')
       end
     end
@@ -139,9 +149,9 @@ module Historiographer
     def historiographer_changes?
       case Rails.version.to_f
       when 0..5 then changed? && valid?
-      when 5.1..7 then saved_changes?
-      else
         raise 'Unsupported Rails version'
+      when 5.1..8 then saved_changes?
+      else
       end
     end
 
@@ -151,17 +161,36 @@ module Historiographer
     # then record history after successful save.
     #
     def should_record_history?
+      return false if snapshot_mode?
+      return false if is_history_class?
+
       historiographer_changes? && !@no_history
     end
 
-    attr_accessor :history_user_id
+    def history_user_id=(value)
+      if is_history_class?
+        write_attribute(:history_user_id, value)
+      else
+        @history_user_id = value
+      end
+    end
+
+    def history_user_id
+      if is_history_class?
+        read_attribute(:history_user_id)
+      else
+        @history_user_id
+      end
+    end
 
     class_name = "#{base.name}History"
 
     begin
       class_name.constantize
     rescue StandardError
-      history_class_initializer = Class.new(ActiveRecord::Base) do
+      history_class_initializer = Class.new(base) do
+        self.table_name = "#{base.table_name}_histories"
+        self.inheritance_column = nil
       end
 
       Object.const_set(class_name, history_class_initializer)
@@ -169,6 +198,70 @@ module Historiographer
 
     klass = class_name.constantize
 
+    # Hook into the association building process
+    base.singleton_class.prepend(Module.new do
+      def belongs_to(name, scope = nil, **options, &extension)
+        super
+        define_history_association(name, :belongs_to, options)
+      end
+
+      def has_one(name, scope = nil, **options, &extension)
+        super
+        define_history_association(name, :has_one, options)
+      end
+
+      def has_many(name, scope = nil, **options, &extension)
+        super
+        define_history_association(name, :has_many, options)
+      end
+
+      def has_and_belongs_to_many(name, scope = nil, **options, &extension)
+        super
+        define_history_association(name, :has_and_belongs_to_many, options)
+      end
+
+      private
+
+      def define_history_association(name, type, options)
+        return if is_history_class?
+        return if @defining_association
+        return if %i[histories current_history].include?(name)
+        @defining_association = true
+
+        history_class = "#{self.name}History".constantize
+        history_class_name = "#{name.to_s.singularize.camelize}History"
+
+        # Get the original association's foreign key
+        original_reflection = self.reflect_on_association(name)
+        foreign_key = original_reflection.foreign_key
+
+        if type == :has_many || type == :has_and_belongs_to_many
+          history_class.send(
+            type, 
+            name, 
+            -> (owner) { where("#{name.to_s.singularize}_histories.snapshot_id = ?", owner.snapshot_id) }, 
+            **options.merge(
+              class_name: history_class_name, 
+              foreign_key: foreign_key,
+              primary_key: foreign_key
+            )
+          )
+        else
+          history_class.send(
+            type, 
+            name, 
+            -> (owner) { where("#{name}_histories.snapshot_id = ?", owner.snapshot_id) }, 
+            **options.merge(
+              class_name: history_class_name, 
+              foreign_key: foreign_key,
+              primary_key: foreign_key
+            )
+          )
+        end
+        @defining_association = false
+      end
+    end)
+
     if base.respond_to?(:histories)
       raise "#{base} already has histories. Talk to Brett if this is a legit use case."
     else
@@ -219,6 +312,46 @@ module Historiographer
       @no_history = false
     end
 
+    
+    def snapshot(tree = {}, snapshot_id = nil)
+      return if is_history_class?
+
+      without_history_user_id do
+        # Use SecureRandom.uuid instead of timestamp for snapshot_id
+        snapshot_id ||= SecureRandom.uuid
+        history_class = self.class.history_class
+        primary_key = self.class.primary_key
+        foreign_key = history_class.history_foreign_key
+        attrs = attributes.clone
+        existing_snapshot = history_class.where(foreign_key => attrs[primary_key], snapshot_id: snapshot_id)
+        return if existing_snapshot.present?
+
+        null_snapshot = history_class.where(foreign_key => attrs[primary_key], snapshot_id: nil)
+        if null_snapshot.present?
+          null_snapshot.update(snapshot_id: snapshot_id)
+        else
+          record_history(snapshot_id: snapshot_id)
+        end
+
+        # Recursively snapshot associations, avoiding infinite loops
+        self.class.reflect_on_all_associations.each do |association|
+          associated_records = send(association.name).reload
+          Array(associated_records).each do |record|
+            model_name = record.class.name
+            record_id = record.id
+
+            tree[model_name] ||= {}
+            next if tree[model_name][record_id]
+
+            new_tree = tree.deep_dup
+            new_tree[model_name][record_id] = true
+
+            record.snapshot(new_tree, snapshot_id) if record.respond_to?(:snapshot)
+          end
+        end
+      end
+    end
+
     private
 
     def history_user_absent_action
@@ -231,8 +364,8 @@ module Historiographer
     #
     # Find the most recent history, and update its history_ended_at timestamp
     #
-    def record_history
-      history_user_absent_action if history_user_id.nil?
+    def record_history(snapshot_id: nil)
+      history_user_absent_action if history_user_id.nil? && should_alert_history_user_id_present?
 
       attrs = attributes.clone
       history_class = self.class.history_class
@@ -240,30 +373,62 @@ module Historiographer
 
       now = UTC.now
       attrs.merge!(foreign_key => attrs['id'], history_started_at: now, history_user_id: history_user_id)
+      attrs.merge!(snapshot_id: snapshot_id) if snapshot_id.present?
 
       attrs = attrs.except('id')
 
       current_history = histories.where(history_ended_at: nil).order('id desc').limit(1).last
 
       if foreign_key.present? && history_class.present?
-        history_class.create!(attrs)
-        current_history.update!(history_ended_at: now) if current_history.present?
+        history_class.create!(attrs).tap do |history|
+          current_history.update!(history_ended_at: now) if current_history.present?
+        end
       else
         raise 'Need foreign key and history class to save history!'
       end
     end
+
+    def without_history_user_id
+      Thread.current[:skip_history_user_id_validation] = true
+      yield
+    ensure
+      Thread.current[:skip_history_user_id_validation] = false
+    end
   end
 
   class_methods do
+    def is_history_class?
+      name.match?(/History$/)
+    end
     #
     # E.g. SponsoredProductCampaign => SponsoredProductCampaignHistory
     #
     def history_class
-      "#{name}History".constantize
+      if is_history_class?
+        nil
+      else
+        "#{name}History".constantize
+      end
     end
 
     def relation
       super.tap { |r| r.extend Historiographer::Relation }
     end
+
+    def historiographer_mode(mode)
+      @historiographer_mode = mode
+    end
+
+    def get_historiographer_mode
+      @historiographer_mode || Historiographer::Configuration.mode
+    end
+  end
+
+  def is_history_class?
+    self.class.is_history_class?
+  end
+
+  def snapshot_mode?
+    (self.class.get_historiographer_mode.to_sym == :snapshot_only)
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: historiographer
 version: !ruby/object:Gem::Version
-  version: 4.0.0
+  version: 4.1.1
 platform: ruby
 authors:
 - brettshollenberger
@@ -228,20 +228,10 @@ extra_rdoc_files:
 - LICENSE.txt
 - README.md
 files:
-- ".document"
-- ".rspec"
-- ".ruby-version"
-- ".standalone_migrations"
-- Gemfile
-- Gemfile.lock
-- Guardfile
 - LICENSE.txt
 - README.md
-- Rakefile
-- VERSION
-- historiographer.gemspec
-- init.rb
 - lib/historiographer.rb
+- lib/historiographer/configuration.rb
 - lib/historiographer/history.rb
 - lib/historiographer/history_migration.rb
 - lib/historiographer/history_migration_mysql.rb
@@ -250,24 +240,7 @@ files:
 - lib/historiographer/relation.rb
 - lib/historiographer/safe.rb
 - lib/historiographer/silent.rb
-- spec/db/database.yml
-- spec/db/migrate/20161121212228_create_posts.rb
-- spec/db/migrate/20161121212229_create_post_histories.rb
-- spec/db/migrate/20161121212230_create_authors.rb
-- spec/db/migrate/20161121212231_create_author_histories.rb
-- spec/db/migrate/20161121212232_create_users.rb
-- spec/db/migrate/20171011194624_create_safe_posts.rb
-- spec/db/migrate/20171011194715_create_safe_post_histories.rb
-- spec/db/migrate/20191024142304_create_thing_with_compound_index.rb
-- spec/db/migrate/20191024142352_create_thing_with_compound_index_history.rb
-- spec/db/migrate/20191024203106_create_thing_without_history.rb
-- spec/db/migrate/20221018204220_create_silent_posts.rb
-- spec/db/migrate/20221018204255_create_silent_post_histories.rb
-- spec/db/schema.rb
-- spec/examples.txt
-- spec/factories/post.rb
-- spec/historiographer_spec.rb
-- spec/spec_helper.rb
+- lib/historiographer/version.rb
 homepage: http://github.com/brettshollenberger/historiographer
 licenses:
 - MIT