data_shifter 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6e4e5f5aa36cfac3275fcb493a2555e6f093873c310395e92d4c7c64c42bb63b
-  data.tar.gz: d7c3e9a682d237887960a0bb3946e73a19bd913709ffb7c069a17a89e45f876a
+  metadata.gz: 8143cec17a5f8cb7374ad327338e694cea0a9422bf0392e005a3eaeeee9ab83d
+  data.tar.gz: b9a246478df8ef89377482951e74ad36b63655510f4bf3bbc6a1b4480edec85e
 SHA512:
-  metadata.gz: 74d1ab829e7d3a695d934f6d624fa6bdad25b7bf13ee11d680ecd90ab9ccfbfa8df33eadf9f3239193a246a183db4604691747db7ae5ea12c4548afed547645a
-  data.tar.gz: c70d3e6be2982dc83501349aeee74ae2e3dd476fd2315f46a76b4e2b52e77cfa6f7c1042a7c4f613460880d07382731b2dd24db41d9e85991b54075d6cf76fa0
+  metadata.gz: e8f150f146151d8a82ccc79fd2e31f2ed813ee1b631d0eb9fc5490fa201ed31890a088f24bb7bb086d4158407f5bfb3f969c639dad75d85809c72d44e609172e
+  data.tar.gz: b776d810b0819d216436e169414c9c09c0a0c1f7cc3123f58912fa638675f55c998c3c7be71bae40868c5fddbb031bd42d3b2b8a915491d9e6d792852712405f

data/.husky/pre-commit

@@ -1,4 +1 @@
-#!/bin/sh
-. "$(dirname "$0")/_/husky.sh"
-
 npx lint-staged

data/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+## [Unreleased]
+
+* N/A
+
+## [0.2.0]
+
+### Added
+
+- **Configuration object**: New `DataShifter.configure` block for global settings.
+- **Dry-run rollback for `transaction false`**: Shifts using `transaction false` (or `:none`) now roll back DB changes in dry-run mode, matching the behavior of other transaction modes.
+- **Automatic side-effect guards in dry run**: When a shift runs in dry run mode, HTTP (via WebMock), ActionMailer, ActiveJob, and Sidekiq (if loaded) are now automatically blocked or faked so that unguarded external calls do not run. Restore happens in an `ensure` so state is reverted after the run.
+  - **HTTP**: All outbound requests are blocked unless allowed with the per-shift `allow_external_requests [...]` DSL or global `DataShifter.config.allow_external_requests`.
+  - **ActionMailer**: `perform_deliveries = false` for the duration of the dry run.
+  - **ActiveJob**: Queue adapter set to `:test` for the duration of the dry run.
+  - **Sidekiq**: `Sidekiq::Testing.fake!` for the duration of the dry run (only if `Sidekiq::Testing` is already loaded).
+- Dependency on `webmock` (>= 3.18) for dry-run HTTP blocking.
+- **Log deduplication**: Repeated log messages are now suppressed during shift runs (default: on). First occurrence logs normally; subsequent occurrences are counted and a summary is printed at the end. Configure globally with `config.suppress_repeated_logs` and `config.repeated_log_cap` (default 1000). Override per-shift with `suppress_repeated_logs false`.
+- **Global progress bar default**: `config.progress_enabled` (default `true`) sets the default for all shifts. Per-shift `progress true/false` still overrides.
+- **Global status interval**: `config.status_interval_seconds` (default `nil`) provides a fallback when `STATUS_INTERVAL` env var is not set.
+- **skip! abort behavior**: `skip!` now terminates the current `process_record` (no `return` needed after calling it).
+- **Grouped skip reasons**: Skip reasons are grouped and the top 10 (by count) are shown in the summary and status output instead of logging each skip inline.
+
+## [0.1.0] - Initial release

data/README.md

@@ -1,6 +1,6 @@
 # DataShifter
 
-Rake-backed data migrations (“shifts”) for Rails apps, with **dry run by default**, progress output, and a consistent summary. Define shift classes in `lib/data_shifts/*.rb`; run them as `rake data:shift:<task_name>`.
+Rake-backed data migrations ("shifts") for Rails apps, with **dry run by default**, progress output, and a consistent summary. Define shift classes in `lib/data_shifts/*.rb`; run them as `rake data:shift:<task_name>`.
 
 ## Installation
 
@@ -21,7 +21,7 @@ Generate a shift (optionally scoped to a model):
 
 ```bash
 bin/rails generate data_shift backfill_foo
-bin/rails generate data_shift backfill_users --model=User
+bin/rails generate data_shift backfill_users --model User
 ```
 
 Add  your logic to the generated file in `lib/data_shifts/`.
@@ -33,19 +33,6 @@ rake data:shift:backfill_foo
 COMMIT=1 rake data:shift:backfill_foo
 ```
 
-## How shift files map to rake tasks
-
-DataShifter defines one rake task per file in `lib/data_shifts/*.rb`.
-
-- **Task name**: derived from the filename with any leading digits removed.
-  - `20260201120000_backfill_foo.rb` → `data:shift:backfill_foo` (leading `<digits>_` prefix is stripped)
-  - `backfill_foo.rb` → `data:shift:backfill_foo`
-- **Class name**: task name camelized, inside the `DataShifts` module.
-  - `backfill_foo` → `DataShifts::BackfillFoo`
-
-Shift files are **required only when the task runs** (tasks are defined up front; classes load lazily).
-The `description "..."` line is extracted from the file and used for `rake -T` output without loading the shift class.
-
 ## Defining a shift
 
 Typical shifts implement:
@@ -77,7 +64,39 @@ Shifts run in **dry run** mode by default. In the automatic transaction modes (`
 - **Commit**: `COMMIT=1 rake data:shift:backfill_foo`
   - (`COMMIT=true` or `DRY_RUN=false` also commit)
 
-Non-DB side effects (API calls, emails, enqueued jobs, etc.) obviously cannot be automatically rolled back, so guard them with e.g. `return if dry_run?`.
+### Automatic side-effect guards (dry run)
+
+In **dry run** mode, DataShifter automatically blocks or fakes these side effects so unguarded code is less likely to hit the network or send mail/jobs:
+
+| Service      | Behavior in dry run |
+|-------------|----------------------|
+| **HTTP**    | Blocked via WebMock (`disable_net_connect!`). Allow specific hosts with `allow_external_requests [...]` or `DataShifter.config.allow_external_requests`. |
+| **ActionMailer** | `perform_deliveries = false` (restored after run). |
+| **ActiveJob**    | Queue adapter set to `:test` (restored after run). |
+| **Sidekiq**      | `Sidekiq::Testing.fake!` (restored with `disable!` after run). Only applied if `Sidekiq::Testing` is already loaded. |
+
+**Guarding other side effects:** For anything we don't cover (e.g. another service, or allowed HTTP that mutates), use e.g. `return if dry_run?` in your shift. DB changes are always rolled back in dry run; only non-DB side effects need this.
+
+To allow HTTP to specific hosts during dry run (e.g. a migration that must call an API to compute values), use the per-shift DSL or global config (NOTE: it is your responsibility to ensure you only make readonly requests in `dry_run?` mode):
+
+```ruby
+# Per shift
+module DataShifts
+  class BackfillFromApi < DataShifter::Shift
+    allow_external_requests ["api.readonly.example.com", %r{\.internal\.company\z}]
+    # ...
+  end
+end
+```
+
+```ruby
+# Global (e.g. in config/initializers/data_shifter.rb)
+DataShifter.configure do |config|
+  config.allow_external_requests = ["api.readonly.example.com"]
+end
+```
+
+Allowed hosts are combined (per-shift + global). Restore (WebMock, mail, jobs) happens in an `ensure` so later code and other specs are unaffected.
 
 ## Transaction modes
 
@@ -85,7 +104,7 @@ Set the transaction mode at the class level:
 
 - **`transaction :single` / `transaction true` (default)**: one DB transaction for the entire run; dry run rolls back at the end; a record error aborts the run.
 - **`transaction :per_record`**: in commit mode, each record runs in its own transaction (errors are collected and the run continues); in dry run, the run is wrapped in a single rollback transaction.
-- **`transaction false` / `transaction :none`**: CAUTION: NOT RECOMMENDED. No automatic transactions and no automatic rollback; ⚠️ **you must manually guard DB writes AND side effects with `dry_run?`.**
+- **`transaction false` / `transaction :none`**: No automatic transaction in **commit** mode only. In dry run, the run is still wrapped in a single rollback transaction so DB changes are never committed. Use when you have external side effects or your own transaction strategy in commit mode.
 
 ```ruby
 module DataShifts
@@ -137,7 +156,53 @@ CONTINUE_FROM=123 COMMIT=1 rake data:shift:backfill_foo
 Notes:
 
 - Only supported for `ActiveRecord::Relation` collections (Array-based collections—like those from `find_exactly!`—cannot be resumed).
-- The filter is `primary_key > CONTINUE_FROM`, so it’s only useful with monotonically increasing primary keys (e.g. `find_each`'s default behavior).
+- The filter is `primary_key > CONTINUE_FROM`, so it's only useful with monotonically increasing primary keys (e.g. `find_each`'s default behavior).
+
+## How shift files map to rake tasks
+
+DataShifter defines one rake task per file in `lib/data_shifts/*.rb`.
+
+- **Task name**: derived from the filename with any leading digits removed.
+  - `20260201120000_backfill_foo.rb` → `data:shift:backfill_foo` (leading `<digits>_` prefix is stripped)
+  - `backfill_foo.rb` → `data:shift:backfill_foo`
+- **Class name**: task name camelized, inside the `DataShifts` module.
+  - `backfill_foo` → `DataShifts::BackfillFoo`
+
+Shift files are **required only when the task runs** (tasks are defined up front; classes load lazily).
+The `description "..."` line is extracted from the file and used for `rake -T` output without loading the shift class.
+
+## Configuration
+
+Configure DataShifter globally in an initializer:
+
+```ruby
+# config/initializers/data_shifter.rb
+DataShifter.configure do |config|
+  # Hosts allowed for HTTP during dry run only (no effect in commit mode)
+  config.allow_external_requests = ["api.readonly.example.com"]
+
+  # Suppress repeated log messages during a shift run (default: true)
+  config.suppress_repeated_logs = true
+
+  # Max unique messages to track for deduplication (default: 1000)
+  config.repeated_log_cap = 1000
+
+  # Global default for progress bar visibility (default: true)
+  config.progress_enabled = true
+
+  # Default status print interval in seconds when ENV STATUS_INTERVAL is not set (default: nil)
+  config.status_interval_seconds = nil
+end
+```
+
+Per-shift overrides:
+
+```ruby
+class MyShift < DataShifter::Shift
+  progress false                # Disable progress bar for this shift
+  suppress_repeated_logs false  # Disable log deduplication for this shift
+end
+```
 
 ## Operational tips
 
@@ -145,7 +210,7 @@ Notes:
 
 - **Start with a dry run**: run the task once with no environment variables set, confirm logs and summary look right, then re-run with `COMMIT=1`.
 - **Make shifts idempotent**: structure `process_record` so re-running is safe (for example, update only when the target column is `NULL`, or compute the same derived value deterministically).
-- **Guard side effects explicitly**: even in dry run, API calls / emails / enqueues are not rolled back. Use `dry_run?` helper to skip side-effectful code.
+- **Guard side effects we don't auto-block**: use `return if dry_run?` for any side effect not covered by Automatic side-effect guards (see above).
 
 ### Choosing a transaction mode (behavior + guidance)
 
@@ -156,8 +221,8 @@ Notes:
   - **Behavior**: in commit mode, records are committed one-by-one; errors are collected and the run continues; the overall run fails at the end if any record failed.
   - **Use when**: you want maximum progress and are OK investigating/fixing a subset of failures.
 - **`transaction false` / `:none`**:
-  - **Behavior**: no automatic transaction wrapper (even in dry run) and no automatic rollback.
-  - **Use when**: you have intentional external side effects, or you’re doing your own transaction/locking strategy—**but always guard writes/side effects with `dry_run?`.**
+  - **Behavior**: in commit mode, no automatic transaction; in dry run, the run is still wrapped in a rollback transaction so DB changes are not committed.
+  - **Use when**: you have intentional external side effects or your own transaction/locking strategy in commit mode.
 
 ### Performance and operability (recommended)
 
@@ -182,17 +247,19 @@ def process_record(buyback)
 end
 ```
 
-### `skip!` (count but don’t update)
+### `skip!` (count but don't update)
 
-Mark a record as skipped (it will increment “Skipped” in the summary):
+Mark a record as skipped. Calling `skip!` terminates the current `process_record` immediately (no `return` needed). The record is counted as "Skipped" in the summary.
 
 ```ruby
 def process_record(record)
   skip!("already done") if record.foo.present?
-  record.update!(foo: value)
+  record.update!(foo: value)  # not executed if skipped
 end
 ```
 
+Skip reasons are grouped: the summary shows the top 10 reasons by count (e.g. `"already done" (42), "not eligible" (3)`) instead of logging each skip inline. This keeps the progress bar clean.
+
 ### Throttling and disabling the progress bar
 
 ```ruby
@@ -202,19 +269,28 @@ class SomeShift < DataShifter::Shift
 end
 ```
 
+
 ## Generator
 
 | Command | Generates |
 |--------|----------|
 | `bin/rails generate data_shift backfill_foo` | `lib/data_shifts/<timestamp>_backfill_foo.rb` with a `DataShifts::BackfillFoo` class |
-| `bin/rails generate data_shift backfill_users --model=User` | Same, with `User.all` in `collection` and `process_record(user)` |
+| `bin/rails generate data_shift backfill_users --model User` | Same, with `User.all` in `collection` and `process_record(user)` |
 | `bin/rails generate data_shift backfill_users --spec` | Also generates `spec/lib/data_shifts/backfill_users_spec.rb` when RSpec is enabled |
 
 The generator refuses to create a second shift if it would produce a duplicate rake task name.
 
 ## Testing shifts (RSpec)
 
-This gem ships a small helper module for running shifts in tests:
+This gem ships a small helper module for running shifts in tests. Require it and include `DataShifter::SpecHelper` in specs or in `RSpec.configure` for `type: :data_shift`.
+
+**Helpers:**
+
+- **`run_data_shift(shift_class, dry_run: true, commit: false)`** — Runs the shift; returns an `Axn::Result`. Use `commit: true` to run in commit mode.
+- **`silence_data_shift_output`** — Suppresses STDOUT for the block (e.g. progress bar).
+- **`capture_data_shift_output`** — Runs the block and returns `[result, output_string]` for asserting on printed output.
+
+Use `expect { ... }.not_to change(...)` and `expect { ... }.to change(...)` to assert that data stays unchanged in dry run and changes when committed:
 
 ```ruby
 require "data_shifter/spec_helper"
@@ -222,35 +298,29 @@ require "data_shifter/spec_helper"
 RSpec.describe DataShifts::BackfillFoo do
   include DataShifter::SpecHelper
 
-  before { allow($stdout).to receive(:puts) } # silence shift output
+  before { allow($stdout).to receive(:puts) }
 
   it "does not persist changes in dry run" do
-    result = run_data_shift(described_class, dry_run: true)
-    expect(result).to be_ok
-    # TODO: add some check confirming data is unchanged
+    expect do
+      result = run_data_shift(described_class, dry_run: true)
+      expect(result).to be_ok
+    end.not_to change(Foo, :count)
   end
 
   it "persists changes when committed" do
-    result = run_data_shift(described_class, commit: true)
-    expect(result).to be_ok
-    # TODO: add some check confirming data is changed
+    expect do
+      result = run_data_shift(described_class, commit: true)
+      expect(result).to be_ok
+    end.to change(Foo, :count).by(1)
+    # Or for in-place updates: .to change { record.reload.bar }.from(nil).to("baz")
   end
 end
 ```
 
-## Optional RuboCop cop
-
-If you use `transaction false` / `transaction :none`, you should guard writes and side effects with `dry_run?`. You can help avoid mistakes by linting that the helper is at least called once via the bundled cop:
-
-```yaml
-# .rubocop.yml
-require:
-  - data_shifter/rubocop
-```
-
 ## Requirements
 
 - Ruby ≥ 3.2.1
-- Rails (ActiveRecord, ActiveSupport, Railties) ≥ 6.1
+- Rails (ActiveRecord, ActiveSupport, Railties) ≥ 7.0
 - `axn` (Shift classes include `Axn`)
 - `ruby-progressbar` (for progress bars)
+- `webmock` (for dry-run HTTP blocking; optional allowlist via `allow_external_requests [...]` / `DataShifter.config.allow_external_requests`)

data/lib/data_shifter/configuration.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+module DataShifter
+  # Global configuration for DataShifter.
+  #
+  # Configure via:
+  #   DataShifter.configure do |config|
+  #     config.allow_external_requests = ["api.readonly.example.com"]
+  #     config.suppress_repeated_logs = true
+  #   end
+  #
+  # Or access directly:
+  #   DataShifter.config.progress_enabled = false
+  class Configuration
+    # Hosts or regexes allowed for HTTP during dry run only (combined with per-shift allow_external_requests).
+    # Has no effect in commit mode — HTTP is unrestricted when dry_run is false.
+    attr_accessor :allow_external_requests
+
+    # Whether to suppress repeated log messages during a shift run. Default: true.
+    # Can be overridden per shift with `suppress_repeated_logs true/false`.
+    attr_accessor :suppress_repeated_logs
+
+    # Maximum unique log messages to track for deduplication. Default: 1000.
+    # When exceeded, entries with count == 1 are cleared first; repeated entries are kept.
+    attr_accessor :repeated_log_cap
+
+    # Global default for progress bar visibility. Default: true.
+    # Per-shift `progress true/false` overrides this.
+    attr_accessor :progress_enabled
+
+    # Default status print interval in seconds when ENV STATUS_INTERVAL is not set. Default: nil.
+    attr_accessor :status_interval_seconds
+
+    def initialize
+      @allow_external_requests = []
+      @suppress_repeated_logs = true
+      @repeated_log_cap = 1000
+      @progress_enabled = true
+      @status_interval_seconds = nil
+    end
+  end
+end

data/lib/data_shifter/errors.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module DataShifter
+  # Raised when a dry run attempts an outbound HTTP request to a host that is
+  # not allowed via allow_external_requests (per-shift or global config).
+  class ExternalRequestNotAllowedError < StandardError
+    def initialize(attempted_host: nil)
+      @attempted_host = attempted_host
+      super(build_message)
+    end
+
+    attr_reader :attempted_host
+
+    private
+
+    def build_message
+      intro = if @attempted_host && !@attempted_host.to_s.strip.empty?
+                "Dry run blocked an outbound HTTP request to #{@attempted_host}."
+              else
+                "Dry run blocked an outbound HTTP request."
+              end
+
+      if @attempted_host && !@attempted_host.to_s.strip.empty?
+        <<~MSG.strip
+          #{intro}
+
+          To allow this host during dry run, add to your shift class:
+
+            allow_external_requests ["#{@attempted_host}"]
+
+          Or set DataShifter.config.allow_external_requests in an initializer.
+        MSG
+      else
+        <<~MSG.strip
+          #{intro}
+
+          To allow specific hosts during dry run, add to your shift class:
+
+            allow_external_requests ["host.example.com"]  # or use a regex
+
+          Or set DataShifter.config.allow_external_requests in an initializer.
+        MSG
+      end
+    end
+  end
+end

data/lib/data_shifter/internal/env.rb

@@ -18,14 +18,16 @@ module DataShifter
         end
       end
 
-      # Parse STATUS_INTERVAL environment variable.
-      # Returns nil if not set or invalid.
+      # Parse STATUS_INTERVAL environment variable, falling back to config.
+      # Returns nil if not set/invalid and config is nil.
       def status_interval_seconds
-        return nil unless ENV["STATUS_INTERVAL"].present?
-
-        Integer(ENV.fetch("STATUS_INTERVAL", nil), 10)
+        if ENV["STATUS_INTERVAL"].present?
+          Integer(ENV.fetch("STATUS_INTERVAL", nil), 10)
+        else
+          DataShifter.config.status_interval_seconds
+        end
       rescue ArgumentError
-        nil
+        DataShifter.config.status_interval_seconds
       end
 
       # Get CONTINUE_FROM environment variable value.

data/lib/data_shifter/internal/log_deduplicator.rb

@@ -0,0 +1,149 @@
+# frozen_string_literal: true
+
+require "digest"
+require "logger"
+
+module DataShifter
+  module Internal
+    # A proxy logger that suppresses repeated log messages during a shift run.
+    # Uses a hash of the message as the key for memory efficiency.
+    # First occurrence is forwarded; subsequent occurrences are counted but not forwarded.
+    # At the end, prints a summary of suppressed messages via puts.
+    class LogDeduplicator
+      attr_reader :real_logger, :cap, :seen
+
+      def initialize(real_logger, cap:)
+        @real_logger = real_logger
+        @cap = cap
+        @seen = {}
+      end
+
+      def add(severity, message = nil, progname = nil, &block)
+        msg = block ? block.call : message
+        key = message_key(severity, progname, msg)
+
+        if @seen.key?(key)
+          @seen[key][:count] += 1
+          nil
+        else
+          enforce_cap
+          @seen[key] = { count: 1, message: truncate_message(msg || progname), severity: }
+          @real_logger.add(severity, message, progname, &block)
+        end
+      end
+
+      def debug(message = nil, progname = nil, &)
+        add(Logger::DEBUG, message, progname, &)
+      end
+
+      def info(message = nil, progname = nil, &)
+        add(Logger::INFO, message, progname, &)
+      end
+
+      def warn(message = nil, progname = nil, &)
+        add(Logger::WARN, message, progname, &)
+      end
+
+      def error(message = nil, progname = nil, &)
+        add(Logger::ERROR, message, progname, &)
+      end
+
+      def fatal(message = nil, progname = nil, &)
+        add(Logger::FATAL, message, progname, &)
+      end
+
+      def unknown(message = nil, progname = nil, &)
+        add(Logger::UNKNOWN, message, progname, &)
+      end
+
+      def <<(msg)
+        key = message_key(Logger::INFO, nil, msg)
+        if @seen.key?(key)
+          @seen[key][:count] += 1
+        else
+          enforce_cap
+          @seen[key] = { count: 1, message: truncate_message(msg), severity: Logger::INFO }
+          @real_logger << msg
+        end
+      end
+
+      def level
+        @real_logger.level
+      end
+
+      def level=(val)
+        @real_logger.level = val
+      end
+
+      def formatter
+        @real_logger.formatter
+      end
+
+      def formatter=(val)
+        @real_logger.formatter = val
+      end
+
+      def close
+        @real_logger.close
+      end
+
+      def suppressed_messages
+        @seen.select { |_k, v| v[:count] > 1 }
+      end
+
+      def print_summary
+        suppressed = suppressed_messages
+        return if suppressed.empty?
+
+        puts "\n[DataShifter] Suppressed repeated log messages:"
+        suppressed.each_value do |entry|
+          count = entry[:count] - 1
+          snippet = entry[:message].to_s[0, 100]
+          snippet = "#{snippet}..." if entry[:message].to_s.length > 100
+          puts "  #{count}x suppressed: #{snippet.inspect}"
+        end
+      end
+
+      def method_missing(method, ...)
+        @real_logger.send(method, ...)
+      end
+
+      def respond_to_missing?(method, include_private = false)
+        @real_logger.respond_to?(method, include_private) || super
+      end
+
+      class << self
+        def with_deduplicating_logger(real_logger, cap:)
+          proxy = new(real_logger, cap:)
+          yield proxy
+        ensure
+          proxy&.print_summary
+        end
+      end
+
+      private
+
+      def message_key(severity, progname, message)
+        normalized = "#{severity}:#{progname}:#{message}"
+        Digest::SHA256.hexdigest(normalized)
+      end
+
+      def truncate_message(msg)
+        str = msg.to_s
+        str.length > 200 ? "#{str[0, 200]}..." : str
+      end
+
+      def enforce_cap
+        return if @seen.size < @cap
+
+        singles = @seen.select { |_k, v| v[:count] == 1 }
+        singles.each_key { |k| @seen.delete(k) } if singles.any?
+
+        return unless @seen.size >= @cap
+
+        oldest_key = @seen.keys.first
+        @seen.delete(oldest_key)
+      end
+    end
+  end
+end

data/lib/data_shifter/internal/output.rb

@@ -11,6 +11,8 @@ module DataShifter
         none: "none",
       }.freeze
 
+      SKIP_REASONS_DISPLAY_LIMIT = 10
+
       module_function
 
       def print_header(io:, shift_class:, total:, label:, dry_run:, transaction_mode:, status_interval:)
@@ -30,7 +32,7 @@ module DataShifter
         io.puts ""
       end
 
-      def print_summary(io:, stats:, errors:, start_time:, dry_run:, transaction_mode:, interrupted:, task_name:, last_successful_id:)
+      def print_summary(io:, stats:, errors:, start_time:, dry_run:, transaction_mode:, interrupted:, task_name:, last_successful_id:, skip_reasons: {})
         return unless start_time
 
         elapsed = (Time.current - start_time).round(1)
@@ -43,6 +45,7 @@ module DataShifter
         io.puts "Succeeded:   #{stats[:succeeded]}"
         io.puts "Failed:      #{stats[:failed]}"
         io.puts "Skipped:     #{stats[:skipped]}"
+        print_skip_reasons(io:, skip_reasons:) if skip_reasons.any?
 
         print_errors(io:, errors:) if errors.any?
         print_interrupt_warning(io:, transaction_mode:, dry_run:) if interrupted
@@ -52,7 +55,7 @@ module DataShifter
         io.puts "=" * 60
       end
 
-      def print_progress(io:, stats:, errors:, start_time:, status_interval:)
+      def print_progress(io:, stats:, errors:, start_time:, status_interval:, skip_reasons: {})
         return unless start_time
 
         elapsed = (Time.current - start_time).round(1)
@@ -74,6 +77,7 @@ module DataShifter
         io.puts "Succeeded:   #{stats[:succeeded]}"
         io.puts "Failed:      #{stats[:failed]}"
         io.puts "Skipped:     #{stats[:skipped]}"
+        print_skip_reasons(io:, skip_reasons:) if skip_reasons.any?
 
         print_errors(io:, errors:) if errors.any?
 
@@ -85,7 +89,9 @@ module DataShifter
         io.puts ""
         io.puts "ERRORS:"
         errors.each do |err|
-          io.puts "  #{err[:record]}: #{err[:error]}"
+          lines = err[:error].to_s.split("\n")
+          io.puts "  #{err[:record]}: #{lines.first}"
+          lines.drop(1).each { |line| io.puts "    #{line}" }
           err[:backtrace]&.each { |line| io.puts "    #{line}" }
         end
       end
@@ -145,6 +151,14 @@ module DataShifter
           status_tips.join(" or ")
         end
       end
+
+      def print_skip_reasons(io:, skip_reasons:)
+        return if skip_reasons.empty?
+
+        top = skip_reasons.sort_by { |_reason, count| -count }.first(SKIP_REASONS_DISPLAY_LIMIT)
+        formatted = top.map { |reason, count| "\"#{reason}\" (#{count})" }.join(", ")
+        io.puts "             #{formatted}"
+      end
     end
   end
 end

data/lib/data_shifter/internal/side_effect_guards.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+require "uri"
+
+module DataShifter
+  module Internal
+    # Applies and restores side-effect guards during dry runs so that HTTP, mail,
+    # and job enqueues are blocked (or faked) unless explicitly allowed.
+    #
+    # Production impact:
+    # - WebMock: required only when apply_webmock runs (i.e. during a dry run), so commit-only
+    #   production runs never load WebMock. On restore we revert to the previous state (enable!
+    #   or disable!) so e.g. specs that had WebMock enabled are not left with it disabled.
+    # - ActionMailer / ActiveJob / Sidekiq: no extra loading; we only toggle existing config
+    #   for the duration of the block and restore in ensure, so impact is scoped to the run.
+    module SideEffectGuards
+      class << self
+        # Applies side-effect guards, yields, then restores. Call only when running in dry run.
+        def with_guards(shift_class:, &block)
+          saved = {}
+          apply_guards(shift_class, saved)
+          block.call
+        rescue webmock_net_connect_error => e
+          host = extract_host_from_webmock_message(e.message)
+          raise DataShifter::ExternalRequestNotAllowedError.new(attempted_host: host), cause: e
+        ensure
+          restore_guards(saved) if saved.any?
+        end
+
+        private
+
+        def apply_guards(shift_class, saved)
+          apply_webmock(shift_class, saved)
+          # rubocop:disable Style/CombinableDefined -- parent must be checked first to avoid NameError when constant not loaded
+          apply_action_mailer(saved) if defined?(ActionMailer) && defined?(ActionMailer::Base)
+          apply_active_job(saved) if defined?(ActiveJob) && defined?(ActiveJob::Base)
+          apply_sidekiq(saved) if defined?(Sidekiq) && defined?(Sidekiq::Testing)
+          # rubocop:enable Style/CombinableDefined
+        end
+
+        def apply_webmock(shift_class, saved)
+          if defined?(WebMock)
+            # WebMock already loaded (e.g. in specs); capture so we can restore
+            saved[:webmock_was_enabled] = net_http_webmock_enabled?
+          else
+            require "webmock"
+            saved[:webmock_was_enabled] = false
+          end
+          WebMock.enable!
+          allowed = allowed_net_hosts(shift_class)
+          opts = allowed.any? ? { allow: allowed } : {}
+          WebMock.disable_net_connect!(**opts)
+          saved[:webmock] = true
+        end
+
+        def net_http_webmock_enabled?
+          Net::HTTP.socket_type.to_s.include?("StubSocket")
+        rescue StandardError
+          false
+        end
+
+        def allowed_net_hosts(shift_class)
+          per_shift = shift_class.respond_to?(:_allow_external_requests) ? shift_class._allow_external_requests : []
+          global = DataShifter.config.allow_external_requests
+          Array(per_shift) + Array(global)
+        end
+
+        def webmock_net_connect_error
+          return WebMock::NetConnectNotAllowedError if defined?(WebMock::NetConnectNotAllowedError)
+
+          Class.new(StandardError) # never matched when WebMock not loaded
+        end
+
+        def extract_host_from_webmock_message(message)
+          return nil unless message.is_a?(String)
+
+          # WebMock format: "Unregistered request: GET https://host/path with headers ..."
+          m = message.match(%r{Unregistered request: \w+ (https?://[^\s]+)})
+          return nil unless m
+
+          uri = URI.parse(m[1])
+          uri.host
+        rescue URI::InvalidURIError, ArgumentError
+          nil
+        end
+
+        def apply_action_mailer(saved)
+          saved[:action_mailer_perform_deliveries] = ActionMailer::Base.perform_deliveries
+          ActionMailer::Base.perform_deliveries = false
+        end
+
+        def apply_active_job(saved)
+          saved[:active_job_adapter] = ActiveJob::Base.queue_adapter
+          ActiveJob::Base.queue_adapter = :test
+        end
+
+        def apply_sidekiq(saved)
+          return unless Sidekiq::Testing.respond_to?(:fake!)
+
+          Sidekiq::Testing.fake!
+          saved[:sidekiq] = true
+        end
+
+        def restore_guards(saved)
+          if saved.delete(:webmock)
+            (saved.delete(:webmock_was_enabled) ? WebMock.enable! : WebMock.disable!)
+          end
+
+          ActionMailer::Base.perform_deliveries = saved.delete(:action_mailer_perform_deliveries) if saved.key?(:action_mailer_perform_deliveries)
+
+          ActiveJob::Base.queue_adapter = saved.delete(:active_job_adapter) if saved.key?(:active_job_adapter)
+
+          return unless saved.delete(:sidekiq)
+
+          Sidekiq::Testing.disable!
+        end
+      end
+    end
+  end
+end

data/lib/data_shifter/shift.rb

@@ -7,6 +7,8 @@ require_relative "internal/output"
 require_relative "internal/signal_handler"
 require_relative "internal/record_utils"
 require_relative "internal/progress_bar"
+require_relative "internal/side_effect_guards"
+require_relative "internal/log_deduplicator"
 
 # Base class for data shifts. Dry-run by default, progress bars, transaction modes, consistent summaries.
 #
@@ -30,15 +32,15 @@ require_relative "internal/progress_bar"
 # Running:
 #   - `rake data:shift:backfill_foo` (dry run by default)
 #   - `COMMIT=1 rake data:shift:backfill_foo` (apply changes)
-#   - Or call directly: `MyShift.call(dry_run: false)` (Axn semantics) - but note default location not auto-loaded
+#   - Or technically can call directly: `MyShift.call(dry_run: false)` (Axn semantics) - BUT:
+#     NOTES: default location not auto-loaded, and in general it is strongly recommended to use the rake task.
 #
 # Transaction modes (set at class level with `transaction`):
 #   - `transaction :single` (default): one transaction for the whole run (all-or-nothing).
 #   - `transaction :per_record`: each record in its own transaction.
-#   - `transaction false`: no automatic transactions; guard writes with `return if dry_run?`.
+#   - `transaction false`: no automatic transaction in commit mode; in dry run we still wrap in a rollback transaction.
 #
-# Dry run: In `:single` and `:per_record`, dry_run rolls back DB changes automatically.
-# Non-DB side effects are not rolled back; guard with `return if dry_run?` / `return unless dry_run?`.
+# Dry run: DB changes are always rolled back (we wrap in a transaction and raise Rollback). Guard non-DB side effects with `return if dry_run?`.
 #
 # Fixed list of IDs (fail fast): Use find_exactly!(Model, [id1, id2, ...]) in `collection`.
 # Large collections: Return an ActiveRecord::Relation and iteration uses `find_each`.
@@ -51,16 +53,25 @@ module DataShifter
 
     log_calls false if respond_to?(:log_calls)
 
+    around :_with_log_deduplication
+    around :_with_side_effect_guards
     around :_with_transaction_for_dry_run
     before :_reset_tracking
     on_success :_print_summary
     on_error :_print_summary
 
     class_attribute :_transaction_mode, default: :single
-    class_attribute :_progress_enabled, default: true
+    class_attribute :_progress_enabled, default: nil
     class_attribute :_description, default: nil
     class_attribute :_task_name, default: nil
     class_attribute :_throttle_interval, default: nil
+    class_attribute :_allow_external_requests, default: [], instance_accessor: false
+    class_attribute :_suppress_repeated_logs, default: nil, instance_accessor: false
+
+    # Internal exception used by skip! to abort the current process_record.
+    # Rescued in _process_one; not propagated.
+    class SkipRecord < StandardError; end
+    private_constant :SkipRecord
 
     class << self
       def description(text = nil)
@@ -104,6 +115,19 @@ module DataShifter
         self._throttle_interval = interval
       end
 
+      # Allow these hosts (or regexes) for HTTP during dry run only. Combines with DataShifter.config.allow_external_requests.
+      # Has no effect in commit mode — HTTP is unrestricted when dry_run is false.
+      # Example: allow_external_requests ["api.readonly.example.com", %r{\.internal\.company\z}]
+      def allow_external_requests(hosts)
+        self._allow_external_requests = Array(hosts)
+      end
+
+      # Enable/disable log deduplication for this shift. Overrides DataShifter.config.suppress_repeated_logs.
+      # Example: suppress_repeated_logs false
+      def suppress_repeated_logs(enabled)
+        self._suppress_repeated_logs = !!enabled
+      end
+
       def run!
         dry_run = Internal::Env.dry_run?
         result = call(dry_run:)
@@ -133,8 +157,9 @@ module DataShifter
 
     def skip!(reason = nil)
       @stats[:skipped] += 1
-      @stats[:succeeded] -= 1
-      log "  SKIP: #{reason}" if reason
+      key = reason.to_s.presence || "(no reason given)"
+      @skip_reasons[key] += 1
+      raise SkipRecord
     end
 
     def log(message)
@@ -145,24 +170,61 @@ module DataShifter
 
     # --- Axn lifecycle hooks ---
 
+    def _with_log_deduplication(chain)
+      effective = self.class._suppress_repeated_logs.nil? ? DataShifter.config.suppress_repeated_logs : self.class._suppress_repeated_logs
+      unless effective && defined?(::Rails) && ::Rails.respond_to?(:logger) && ::Rails.logger
+        chain.call
+        return
+      end
+
+      original_logger = ::Rails.logger
+      original_ar_logger = ::ActiveRecord::Base.logger
+
+      Internal::LogDeduplicator.with_deduplicating_logger(original_logger, cap: DataShifter.config.repeated_log_cap) do |proxy|
+        ::Rails.logger = proxy
+        ::ActiveRecord::Base.logger = proxy
+        chain.call
+      end
+    ensure
+      if effective && defined?(::Rails) && ::Rails.respond_to?(:logger=)
+        ::Rails.logger = original_logger
+        ::ActiveRecord::Base.logger = original_ar_logger
+      end
+    end
+
+    def _with_side_effect_guards(chain)
+      if dry_run?
+        Internal::SideEffectGuards.with_guards(shift_class: self.class) { chain.call }
+      else
+        chain.call
+      end
+    end
+
     def _with_transaction_for_dry_run(chain)
       if _transaction_mode == :none
-        chain.call
+        if dry_run?
+          ::ActiveRecord::Base.transaction do
+            chain.call
+            raise ::ActiveRecord::Rollback
+          end
+        else
+          chain.call
+        end
         return
       end
 
       if _transaction_mode == :single
-        ActiveRecord::Base.transaction do
+        ::ActiveRecord::Base.transaction do
           chain.call
-          raise ActiveRecord::Rollback if dry_run?
+          raise ::ActiveRecord::Rollback if dry_run?
         end
         return
       end
 
       if dry_run?
-        ActiveRecord::Base.transaction do
+        ::ActiveRecord::Base.transaction do
           chain.call
-          raise ActiveRecord::Rollback
+          raise ::ActiveRecord::Rollback
         end
       else
         chain.call
@@ -172,6 +234,7 @@ module DataShifter
     def _reset_tracking
       @stats = { processed: 0, succeeded: 0, failed: 0, skipped: 0 }
       @errors = []
+      @skip_reasons = Hash.new(0)
       @start_time = Time.current
       @last_status_print = @start_time
       @_data_shift_interrupted = false
@@ -183,6 +246,7 @@ module DataShifter
         io: $stdout,
         stats: @stats,
         errors: @errors,
+        skip_reasons: @skip_reasons,
         start_time: @start_time,
         dry_run: dry_run?,
         transaction_mode: _transaction_mode,
@@ -209,6 +273,7 @@ module DataShifter
         io: $stdout,
         stats: @stats,
         errors: @errors,
+        skip_reasons: @skip_reasons,
         start_time: @start_time,
         status_interval: Internal::Env.status_interval_seconds,
       )
@@ -275,18 +340,19 @@ module DataShifter
     # --- Transaction execution strategies ---
 
     def _run_in_single_transaction(enum, total, &block)
-      ActiveRecord::Base.transaction do
+      ::ActiveRecord::Base.transaction do
         _iterate(enum, total, &block)
         if dry_run?
           log "\nDry run complete — rolling back all changes."
-          raise ActiveRecord::Rollback
+          raise ::ActiveRecord::Rollback
         end
       end
     rescue StandardError => e
       return if @errors.any?
 
       @stats[:failed] += 1
-      @errors << { record: "transaction", error: e.message, backtrace: e.backtrace&.first(3) }
+      error_text = _format_error(e)
+      @errors << { record: "transaction", error: error_text, backtrace: e.backtrace&.first(3) }
     end
 
     def _run_per_record(enum, total, &)
@@ -294,7 +360,7 @@ module DataShifter
         if dry_run?
           yield record
         else
-          ActiveRecord::Base.transaction { yield record }
+          ::ActiveRecord::Base.transaction { yield record }
         end
       end
     end
@@ -304,7 +370,8 @@ module DataShifter
     end
 
     def _iterate(enum, total)
-      bar = Internal::ProgressBar.create(total:, dry_run: dry_run?, enabled: _progress_enabled)
+      progress_on = _progress_enabled.nil? ? DataShifter.config.progress_enabled : _progress_enabled
+      bar = Internal::ProgressBar.create(total:, dry_run: dry_run?, enabled: progress_on)
       if enum.respond_to?(:find_each)
         enum.find_each do |record|
           _process_one(record) { yield record }
@@ -325,11 +392,15 @@ module DataShifter
       yield
       @stats[:succeeded] += 1
       @_last_successful_id = record.id if record.respond_to?(:id)
+    rescue SkipRecord
+      # skip! already incremented @stats[:skipped] and recorded the reason; just continue
+      nil
     rescue StandardError => e
       @stats[:failed] += 1
       identifier = Internal::RecordUtils.identifier(record)
-      @errors << { record: identifier, error: e.message, backtrace: e.backtrace&.first(3) }
-      log "ERROR #{identifier}: #{e.message}"
+      error_text = _format_error(e)
+      @errors << { record: identifier, error: error_text, backtrace: e.backtrace&.first(3) }
+      _log_error(identifier, error_text)
 
       raise if _transaction_mode == :single
     ensure
@@ -345,6 +416,18 @@ module DataShifter
       _print_progress
     end
 
+    def _format_error(e)
+      msg = e.message.to_s
+      msg += "\n  Caused by: #{e.cause.class}: #{e.cause.message}" if e.respond_to?(:cause) && e.cause
+      msg
+    end
+
+    def _log_error(identifier, error_text)
+      lines = error_text.to_s.split("\n")
+      log "ERROR #{identifier}: #{lines.first}"
+      lines.drop(1).each { |line| log "    #{line}" }
+    end
+
     # --- Output helpers ---
 
     def _print_header(total)

data/lib/data_shifter/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DataShifter
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/data_shifter.rb

@@ -1,5 +1,26 @@
 # frozen_string_literal: true
 
 require_relative "data_shifter/version"
+require_relative "data_shifter/configuration"
+require_relative "data_shifter/errors"
 require_relative "data_shifter/shift"
 require_relative "data_shifter/railtie"
+
+module DataShifter
+  class << self
+    # Returns the global configuration instance.
+    def config
+      @config ||= Configuration.new
+    end
+
+    # Yields the configuration for block-style setup.
+    #
+    #   DataShifter.configure do |c|
+    #     c.allow_external_requests = ["api.readonly.example.com"]
+    #     c.suppress_repeated_logs = false
+    #   end
+    def configure
+      yield config
+    end
+  end
+end

data/lib/generators/data_shift_generator.rb

@@ -83,6 +83,7 @@ class DataShiftGenerator < Rails::Generators::NamedBase
     underscored_name = name.underscore
     record_arg = @model_name.present? ? @model_name.underscore : "record"
 
+    model_for_change = @model_name.present? ? @model_name : "Model"
     create_file "spec/lib/data_shifts/#{underscored_name}_spec.rb", <<~RUBY
       # frozen_string_literal: true
 
@@ -94,22 +95,24 @@ class DataShiftGenerator < Rails::Generators::NamedBase
 
         before { allow($stdout).to receive(:puts) }
 
-        # TODO: Set up test records
+        # Set up test records as needed
         # let(:#{record_arg}) { create(:#{record_arg}) }
 
         describe "dry run" do
           it "does not persist changes" do
-            result = run_data_shift(described_class, dry_run: true)
-            expect(result).to be_ok
-            # TODO: Assert that records are unchanged
+            expect do
+              result = run_data_shift(described_class, dry_run: true)
+              expect(result).to be_ok
+            end.not_to change(#{model_for_change}, :count)
           end
         end
 
         describe "commit" do
           it "applies changes" do
-            result = run_data_shift(described_class, commit: true)
-            expect(result).to be_ok
-            # TODO: Assert that records are updated
+            expect do
+              result = run_data_shift(described_class, commit: true)
+              expect(result).to be_ok
+            end.to change(#{model_for_change}, :count)
           end
         end
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_shifter
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Kali Donovan
@@ -85,6 +85,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '1.13'
+- !ruby/object:Gem::Dependency
+  name: webmock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.18'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.18'
 description: 'DataShifter: backfills and one-off fixes as rake tasks. Dry run by default,
   auto rollback, progress bars, consistent summaries.'
 email:
@@ -95,22 +109,25 @@ extra_rdoc_files: []
 files:
 - ".husky/pre-commit"
 - ".lintstagedrc"
+- CHANGELOG.md
 - LICENSE.txt
 - README.md
 - Rakefile
 - lib/data_shifter.rb
+- lib/data_shifter/configuration.rb
+- lib/data_shifter/errors.rb
 - lib/data_shifter/internal/env.rb
+- lib/data_shifter/internal/log_deduplicator.rb
 - lib/data_shifter/internal/output.rb
 - lib/data_shifter/internal/progress_bar.rb
 - lib/data_shifter/internal/record_utils.rb
+- lib/data_shifter/internal/side_effect_guards.rb
 - lib/data_shifter/internal/signal_handler.rb
 - lib/data_shifter/railtie.rb
-- lib/data_shifter/rubocop.rb
 - lib/data_shifter/shift.rb
 - lib/data_shifter/spec_helper.rb
 - lib/data_shifter/version.rb
 - lib/generators/data_shift_generator.rb
-- lib/rubocop/cop/data_shifter/skip_transaction_guard_dry_run.rb
 homepage: https://github.com/teamshares/data_shifter
 licenses:
 - MIT

data/lib/data_shifter/rubocop.rb

@@ -1,4 +0,0 @@
-# frozen_string_literal: true
-
-require "rubocop"
-require "rubocop/cop/data_shifter/skip_transaction_guard_dry_run"

data/lib/rubocop/cop/data_shifter/skip_transaction_guard_dry_run.rb

@@ -1,55 +0,0 @@
-# frozen_string_literal: true
-
-module RuboCop
-  module Cop
-    module DataShifter
-      # In data shift files, `transaction false` disables automatic transaction
-      # and rollback. DB writes (and side effects) are not rolled back on dry run, so
-      # the shift must guard them with `return if dry_run?` or `return unless dry_run?`.
-      #
-      # @example
-      #   # bad
-      #   class BackfillUsers < DataShifter::Shift
-      #     transaction false
-      #     def process_record(record)
-      #       record.update!(foo: 1)
-      #     end
-      #   end
-      #
-      #   # good
-      #   class BackfillUsers < DataShifter::Shift
-      #     transaction false
-      #     def process_record(record)
-      #       return if dry_run?
-      #       record.update!(foo: 1)
-      #     end
-      #   end
-      class SkipTransactionGuardDryRun < Base
-        MSG = "Data shifts using `transaction false` must guard writes/side effects with " \
-              "`return if dry_run?` or `return unless dry_run?`."
-
-        def_node_matcher :skip_transaction_call?, <<~PATTERN
-          (send _ :transaction {(sym :none) (false)})
-        PATTERN
-
-        def on_send(node)
-          return unless skip_transaction_call?(node)
-          return if file_contains_dry_run_guard?
-
-          add_offense(node, message: MSG)
-        end
-
-        private
-
-        def file_contains_dry_run_guard?
-          return true unless processed_source.ast
-
-          processed_source.ast.each_node(:send) do |send_node|
-            return true if send_node.method?(:dry_run?)
-          end
-          false
-        end
-      end
-    end
-  end
-end