after_migrate 0.2.6 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8c9c1befe1200c2536f27cf80f99dc05d1b825745c1d01c13499dae96275b61f
-  data.tar.gz: 8e8cc6b87918d25303c668c1cbe2fbfd4129c937bac9b83284a7ea639fb42453
+  metadata.gz: 453b86e338e7eb2ba4eb0af92cde363cee1de07ade6bcaaec31f953cc6938e32
+  data.tar.gz: f2673ef800205a658f30d54865724e34b77b73a5fcd6df7e7b608af1d5df3fb7
 SHA512:
-  metadata.gz: 42015b8f9c703839c91318c2bc165ff924f3e0b65dd4a0e6e770721456be25f560ea5674f8b50638130fb9bcce4ea6033aeddad7f4893bf4a9db0fe70a03e1c1
-  data.tar.gz: bae706e4aaaa059698cc9e9fc45c1e4f05c32d0227b1408ba374359d44d3b1d7ab69e4ae9e450747032d0d27381bc394d8659772593525d45d06ef0e5a409ab4
+  metadata.gz: 219bcd2a38ec717a6837024b6c3d969d0e6ec04d23f153ee2ff1eb1a096bcd221b7ec396f428db4906a65afa317ad17abc13f2a3320de5ae1cd64d6c058d09e1
+  data.tar.gz: 290f5f231817b412be23ebe12235ed430fbf7f660e7be20bdb7c8aa5f2cc731bae4a55ffeaae9888fbc3ccfa4722dbf6c9aabe595f1305d8a5d01a7ca31f0a5f

data/.gitignore

@@ -12,3 +12,4 @@
 .idea/
 /Gemfile.lock
 .private/
+.DS_Store

data/AGENTS.md

@@ -0,0 +1,74 @@
+# AGENTS.md
+
+This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
+
+## Commands
+
+```bash
+bundle install          # install dependencies
+bundle exec rspec       # run all tests
+bundle exec rspec spec/after_migrate_spec.rb  # run a single spec file
+bundle exec rake        # default task (runs spec)
+bundle exec rubocop     # lint
+bundle exec rubocop -a  # lint with auto-fix
+rake release            # build and push gem to RubyGems
+```
+
+## Architecture
+
+This is a Rails gem that automatically runs database maintenance (`ANALYZE`, `VACUUM`, `PRAGMA optimize`) after `db:migrate` tasks. The core flow is:
+
+1. **`Railtie`** (`lib/after_migrate/railtie.rb`) — the entry point. On Rails init, it subscribes to `sql.active_record` notifications so every SQL statement during a migration is intercepted. It enhances `db:migrate`, `db:migrate:up`, and `db:migrate:redo` to call `AfterMigrate.run!` when they complete — unless `defer: true` (the default), in which case the rake task only collects. It also registers the `after_migrate:run` task.
+
+2. **`Collector`** (`lib/after_migrate/collector.rb`) — receives every SQL notification, filters to DDL/DML statements (CREATE/ALTER/DROP/INSERT/UPDATE/DELETE), calls the adapter-specific parser, and accumulates table names into `AfterMigrate.affected_tables` (a `Concurrent::Map<schema, Concurrent::Set<table_name>>`).
+
+3. **`AfterMigrate.store`** (`lib/after_migrate/store.rb`) — defaults to an in-memory `Concurrent::Map` wrapped by `AfterMigrate::Stores::Memory`. `AfterMigrate.affected_tables`, `merge_tables`, and `reset!` delegate to the store. The memory store persists across multiple rake task invocations inside one Ruby process and is cleared by `AfterMigrate.run!` (or `AfterMigrate.reset!`). This replaces the old `Current` (`ActiveSupport::CurrentAttributes`) which was reset after every migration. Alternate backends: `FileStore` (JSON, cross-invocation) and `RedisStore` (cross-process, for parallel/multi-tenant runs). `RedisStore#resolved_redis` memoizes the resolved client per store instance, so `config.redis` is only materialized once — pass a `ConnectionPool` (the store calls `pool.with`) or a memoized client, never a `-> { Redis.new }` proc that builds a connection per call (that opens one connection per SQL statement and exhausts the Redis connection limit under concurrency).
+
+4. **`Executor`** (`lib/after_migrate/executor.rb`) — iterates `AfterMigrate.affected_tables` and calls the correct adapter's `optimize_tables`. Respects the `analyze` config option (`only_affected_tables` / `all_tables` / `none`). Always calls `AfterMigrate.reset!` in its `ensure` block.
+
+5. **Adapters** (`lib/after_migrate/adapters/`) — one module per database:
+   - `Sql` — shared regex-based table parser (used by MySQL and SQLite, which can't use pg_query)
+   - `Postgresql` — uses `pg_query` gem for accurate SQL parsing; runs `VACUUM` (checking `pg_stat_all_tables` for dead tuples) then `ANALYZE VERBOSE` per table
+   - `Mysql` — runs `ANALYZE TABLE` per table; lists tables from `information_schema`
+   - `Sqlite` — runs `PRAGMA optimize` (SQLite ≥ 3.35.0) or `VACUUM; ANALYZE;`
+
+## Key design decisions
+
+- PostgreSQL uses `pg_query` (the actual Postgres parser) for table extraction, which avoids false positives from regex. MySQL and SQLite fall back to the shared `Sql` regex patterns.
+- `pg_query` is a hard runtime dependency (listed in gemspec), not optional — even though it's only used for PostgreSQL.
+- Table collection uses `Concurrent::Map` + `Concurrent::Set` (from `concurrent-ruby`) for thread-safe accumulation across parallel migration workers.
+- The gem does **not** monkey-patch ActiveRecord. It only uses public `ActiveSupport::Notifications` and `Rake::Task#enhance` APIs.
+- `defer: true` (the default) is the multi-tenant-friendly mode: rake tasks only collect, never execute. Call `AfterMigrate.run!` (or `rake after_migrate:run`) once after all tenant migrations complete.
+- The `app.executor.to_run` unsubscription in `Railtie` ensures the SQL subscription is dropped before the app starts serving web requests, so normal traffic is never collected.
+
+## Configuration
+
+Config values are in `AfterMigrate::Configuration` (initialized in `lib/after_migrate.rb`):
+
+| Option                | Default                  | Values                                                  |
+|-----------------------|--------------------------|---------------------------------------------------------|
+| `enabled`             | `true`                   | bool                                                    |
+| `verbose`             | `true`                   | bool                                                    |
+| `vacuum`              | `true`                   | bool (PostgreSQL only)                                  |
+| `analyze`             | `"only_affected_tables"` | `"only_affected_tables"`, `"all_tables"`, `"none"`      |
+| `rake_tasks_enhanced` | `true`                   | bool                                                    |
+| `defer`               | `true`                   | bool — skip auto-run; call `AfterMigrate.run!` manually |
+
+## Multi-tenant usage
+
+```ruby
+# In your tenant migration runner:
+Tenant.each do |tenant|
+  tenant.switch { ActiveRecord::MigrationContext.new(...).migrate }
+end
+# Tables are now accumulated per schema in AfterMigrate.affected_tables
+AfterMigrate.run!           # or: Rake::Task['after_migrate:run'].invoke
+```
+
+Set `defer: false` to restore the v0.1 behaviour of running after each `db:migrate`.
+
+## Dependencies
+
+- Ruby ≥ 3.2, Rails ≥ 7.0
+- `pg_query ≥ 6.1` (required even for non-Postgres installs)
+- Dev: `rspec ~> 3.0`, `rubocop ~> 1.81`, `bundler ~> 4`

data/CHANGELOG.md

@@ -2,6 +2,57 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.3.0] - 2026-08-07
+
+### Fixed
+- **Worker/web processes could hang indefinitely.** The railtie registered
+  `app.executor.to_run { ActiveSupport::Notifications.unsubscribe(subscription) }`, which fires on
+  *every* executor invocation - every web request and, through `Sidekiq::Rails::Reloader`, every
+  background job. Each call takes the process-global `ActiveSupport::Notifications::Fanout` mutex and
+  clears the listener cache, forcing every subsequent instrumented statement to contend on the same
+  mutex via `Fanout#all_listeners_for`. Under concurrency this collapsed: a production Sidekiq
+  process lost all five worker threads for 7+ hours, each parked on `fanout.rb:82`
+  `Thread::Mutex#synchronize` inside this callback. The unsubscribe is now one-shot, so every call
+  after the first is a boolean check.
+
+- **Store failures no longer abort migrations.** `Collector` runs inside an `sql.active_record`
+  subscriber, and ActiveSupport propagates subscriber exceptions to whoever emitted the event -- so a
+  Redis error aborted the SQL statement and with it the migration. Running 25 parallel
+  `rails apartment:migrate` workers against one Redis produced
+  `Connection refused - connect(2) for <host>:6379` and killed migrations mid-run. `merge_tables`,
+  `affected_tables` and `reset!` now degrade to a warning: the first failure is logged in full,
+  subsequent ones are counted (`AfterMigrate.store_failure_count`), and `AfterMigrate.store_degraded?`
+  reports that collected tables are incomplete. Losing table names only costs some ANALYZE coverage;
+  a half-applied migration costs far more.
+
+### Added
+- `config.raise_on_store_error` (default `false`) to restore the old fatal behaviour.
+- `AfterMigrate.store_degraded?` and `AfterMigrate.store_failure_count`.
+
+### Changed
+- **BREAKING: `enabled` is now auto-detected**, resolved at `Configuration` construction time:
+
+  | `AFTER_MIGRATE_ENABLED` | result                                        |
+  |-------------------------|-----------------------------------------------|
+  | `true`                  | always on                                     |
+  | `false`                 | always off - an explicit opt-out always wins  |
+  | unset / blank           | **on only while a migration task is running** |
+
+  Auto-detection matches the running task against `db:migrate`, `db:migrate:up`, `db:rollback`,
+  `apartment:migrate`, `db:schema:load`, `db:structure:load` and `after_migrate:*`, reading
+  `Rake.application.top_level_tasks` with `ARGV` as a fallback. It deliberately does not match
+  `db:seed`, `assets:precompile`, `server`, `console`, or a Sidekiq command line -- so a web or
+  worker process never arms the gem, while `rails db:migrate` needs no configuration at all.
+
+  Previously, there was no way to keep the gem out of long-running processes.
+
+  This gem is a migration-time tool, and there was previously no way to keep it out of long-running
+  processes: railtie initializers run *before* `:load_config_initializers`
+  (`Rails::Application#ordered_railties` pushes the application last), so a host app calling
+  `AfterMigrate.configure { |c| c.enabled = false }` in `config/initializers/` was always too late --
+  the railtie had already subscribed and registered the executor hook. Reading the environment
+  variable directly is the only way the railtie can see the intended value.
+
 ## [0.2.6] - 2026-07-16
 
 ### Fixed

data/README.md

@@ -59,10 +59,22 @@ Out of the box, it runs `ANALYZE` on **only the tables touched** during the migr
 
 Create `config/initializers/after_migrate.rb`:
 
+> **`enabled` is auto-detected -- do not set it here.**
+>
+> | `AFTER_MIGRATE_ENABLED` | result                                                                                                          |
+> |-------------------------|-----------------------------------------------------------------------------------------------------------------|
+> | `true`                  | always on                                                                                                       |
+> | `false`                 | always off (explicit opt-out wins)                                                                              |
+> | unset                   | **on only while running `db:migrate`, `db:rollback`, `apartment:migrate`, `db:schema:load`, `after_migrate:*`** |
+>
+> It must be resolved from the environment, because Rails runs railtie initializers *before*
+> `config/initializers/*.rb` -- by the time this block executes, the gem has already decided whether
+> to subscribe. Assigning `config.enabled` here still affects `AfterMigrate.run!`, but **cannot**
+> switch collection on or off, and setting it to a fixed value will override auto-detection for
+> `run!`. Leave it alone unless you mean it.
+
 ```ruby
 AfterMigrate.configure do |config|
-  # Enable/disable the gem
-  config.enabled = true
 
   # Log what’s happening
   config.verbose = true

data/lib/after_migrate/railtie.rb

@@ -12,9 +12,27 @@ module AfterMigrate
         AfterMigrate::Collector.call(*args)
       end
 
-      # Unsubscribe when the app starts serving requests so normal web traffic
-      # is never collected.
-      app.executor.to_run { ActiveSupport::Notifications.unsubscribe(subscription) }
+      # Unsubscribe once, the first time the app starts doing work, so normal web/worker traffic is
+      # never collected.
+      #
+      # THIS MUST STAY ONE-SHOT. `executor.to_run` fires on EVERY executor invocation -- every web
+      # request and, via Sidekiq::Rails::Reloader, every single background job. Calling
+      # `Notifications.unsubscribe` there takes the process-global
+      # `ActiveSupport::Notifications::Fanout` mutex and calls `clear_cache`, which invalidates the
+      # listener cache so every subsequent instrumented statement must take the same mutex through
+      # `Fanout#all_listeners_for`.
+      done = false
+      lock = Mutex.new
+      app.executor.to_run do
+        next if done
+
+        lock.synchronize do
+          next if done
+
+          done = true
+          ActiveSupport::Notifications.unsubscribe(subscription)
+        end
+      end
     end
 
     rake_tasks do

data/lib/after_migrate/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module AfterMigrate
-  VERSION = '0.2.6'
+  VERSION = '0.3.0'
 end

data/lib/after_migrate.rb

@@ -9,15 +9,80 @@ require 'after_migrate/railtie'
 module AfterMigrate
   class Configuration
     attr_accessor :enabled, :verbose, :vacuum, :analyze, :rake_tasks_enhanced, :defer,
-                  :store, :run_id, :store_options
+                  :store, :run_id, :store_options, :raise_on_store_error
+
+    # Task names that mean "this process is applying schema changes". Matches `db:migrate`,
+    # `db:migrate:up`, `db:rollback`, `apartment:migrate`, `db:schema:load`, `after_migrate:*` --
+    # and deliberately NOT `db:seed`, `assets:precompile`, `server`, `console`, or a bare Sidekiq
+    # command line.
+    MIGRATION_TASK_PATTERN = /
+      (?:\A|:)(?:migrate|rollback)(?::|\z)
+      | \Adb:(?:schema|structure):load\z
+      | \Aafter_migrate:
+    /x
+
+    class << self
+      # Three states, so the common case needs no configuration at all:
+      #
+      #   AFTER_MIGRATE_ENABLED=true    -> always on  (force it, e.g. from a console)
+      #   AFTER_MIGRATE_ENABLED=false   -> always off (explicit opt-out always wins)
+      #   unset                         -> ON only while running migrations
+      #
+      # Auto-detection is safe because a web or worker process never has a migration task on its
+      # command line, so the gem simply never arms itself there.
+      def resolve_enabled
+        override = env_override
+        return override unless override.nil?
+
+        migration_task?
+      end
+
+      # nil when unset/blank so "not configured" stays distinguishable from "configured false".
+      def env_override
+        raw = ENV['AFTER_MIGRATE_ENABLED'].to_s.strip
+        return nil if raw.empty?
+
+        raw.casecmp('true').zero?
+      end
+
+      def migration_task?
+        task_names.any? { |name| MIGRATION_TASK_PATTERN.match?(name) }
+      end
+
+      # Rake's parsed task list is the reliable source (`rails db:migrate` dispatches through Rake
+      # and the environment is loaded by the `environment` prerequisite, so top_level_tasks is
+      # already populated by the time our railtie runs). ARGV is the belt-and-braces fallback for
+      # `rake db:migrate` and for anything that bypasses Rake's parser.
+      def task_names
+        names = []
+        if defined?(::Rake) && ::Rake.respond_to?(:application)
+          top_level = ::Rake.application.top_level_tasks
+          names.concat(Array(top_level))
+        end
+        names.concat(Array(ARGV))
+        names.map(&:to_s)
+      rescue StandardError
+        Array(ARGV).map(&:to_s)
+      end
+    end
 
+    # Resolved here, not in a host app's `config/initializers/*.rb`.
+    #
+    # Railtie initializers run BEFORE `:load_config_initializers` (Rails::Application#ordered_railties
+    # pushes the application last), so by the time an app calls `AfterMigrate.configure`, our railtie
+    # has already decided whether to subscribe. An app setting `config.enabled = false` therefore
+    # could NOT switch the subscription off -- the only value the railtie ever sees is this one.
+    #
+    # Off outside migrations: leaving this armed in every web and worker process.
     def initialize
-      @enabled = true
+      @enabled = self.class.resolve_enabled
       @verbose = true
       @vacuum = true
       @analyze = 'only_affected_tables'
       @rake_tasks_enhanced = true
       @defer = true
+      # Store failures degrade to a warning by default; they must not abort a migration.
+      @raise_on_store_error = false
       @store = :memory
       @run_id = ENV.fetch('AFTER_MIGRATE_RUN_ID', 'default')
       @store_options = {
@@ -84,11 +149,21 @@ module AfterMigrate
 
     # Persistent cross-migration store: schema_name => Concurrent::Set<table_name>
     def affected_tables
-      store.affected_tables
+      with_store_rescue('affected_tables', default: Concurrent::Map.new) { store.affected_tables }
     end
 
     def merge_tables(schema, table_names)
-      store.merge_tables(schema, table_names)
+      with_store_rescue('merge_tables') { store.merge_tables(schema, table_names) }
+    end
+
+    # True, when at least one store operation failed and was swallowed, so the collected table list
+    # is incomplete and maintenance coverage will be partial.
+    def store_degraded?
+      @store_degraded == true
+    end
+
+    def store_failure_count
+      @store_failure_count.to_i
     end
 
     # Trigger database maintenance on all collected tables, then clear the store.
@@ -100,7 +175,36 @@ module AfterMigrate
     end
 
     def reset!
-      store.reset!
+      with_store_rescue('reset!') { store.reset! }
+      @store_degraded = false
+      @store_failure_count = 0
+      nil
+    end
+
+    # A maintenance tool must never break the thing it is measuring.
+    #
+    # `Collector` runs inside an `sql.active_record` notification subscriber, and ActiveSupport
+    # propagates subscriber exceptions to the code that emitted the event — so a store error here
+    # aborts the SQL statement, which aborts the migration.
+    #
+    # Losing collected table names only means some tables miss their ANALYZE -- recoverable, and far
+    # cheaper than a half-applied migration. Set `config.raise_on_store_error = true` to opt out.
+    #
+    # The first failure logs in full; the rest are counted only, so a long migration cannot flood
+    # the log with one line per statement.
+    def with_store_rescue(operation, default: nil)
+      yield
+    rescue StandardError => e
+      raise if configuration.raise_on_store_error
+
+      @store_failure_count = store_failure_count + 1
+      unless @store_degraded
+        @store_degraded = true
+        log("store #{operation} failed (#{e.class}: #{e.message}). Continuing without the store — " \
+            'collected tables will be incomplete and some maintenance may be skipped. ' \
+            'Set config.raise_on_store_error = true to make this fatal.')
+      end
+      default
     end
 
     def store

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: after_migrate
 version: !ruby/object:Gem::Version
-  version: 0.2.6
+  version: 0.3.0
 platform: ruby
 authors:
 - Nikolay Moskvin
@@ -52,6 +52,7 @@ files:
 - ".rubocop.yml"
 - ".ruby-version"
 - ".travis.yml"
+- AGENTS.md
 - CHANGELOG.md
 - CLAUDE.md
 - Gemfile