hekenga 2.0.0 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f21c3c1cb0e45c3b9eb2627fc960d032ff981e01b239dc69c02d2aebc1f7b539
-  data.tar.gz: 62fbbd8a65bae8bacc537bcc40cc4a3960b795ebb19a13c34b8ebd539eb102d0
+  metadata.gz: d81f0def52813a113005c985703355b9d207a9fb91acae8e4948d01dc43f7b33
+  data.tar.gz: f462545e892ababf6a251dd5d366ef59cfc0f799b1ba3456102ee0483cab652d
 SHA512:
-  metadata.gz: 6317b298a05085564cfaaee1beef6b5749e83f8bc877538b448db66b50c3dde50f0380a5fa7f178dff7feca6840564c486f263891a838d4ed51617b7ff4f8698
-  data.tar.gz: 12ba1c564acca2f7d43c80a384a0767b00592abf5126d0016a0bf7cadf610e6d342890cd54597a602654e8449491135a3dda488bb7c1c8aaeb72d035bdc7c5d6
+  metadata.gz: 62e018a5ad375a280f641847c3c85c18b190ff7ef83d558a53210b062a80eecb183168af77baac2d2a7f2b7d5d16438b2f6057faaaa7a52e410f952a46673d69
+  data.tar.gz: 10b34cb83cf1f8d643c3cd11f06389ae6911d84c6afa7ff7901a681a735a83bb24108178a162ba0005ef7ed1992c54448ac3eff4479498533505a7ab65a0a71e

data/CHANGELOG.md

@@ -1,8 +1,25 @@
 # Changelog
 
+## v2.2.0
+
+- Added a `hekenga watch <path_or_pkey>` CLI command to attach to a running
+    migration and report its status periodically. Accepts an `--interval`
+    option (defaults to `Hekenga.config.report_sleep`).
+- Added a `hekenga failures <path_or_pkey>` CLI command to print all failed and
+    invalid document IDs across a migration's document tasks.
+- Document tasks now support a `skip_validation!` option to write documents
+    without running `.valid?`.
+
+## v2.1.0
+
+- `per_document` task `scope` will no longer let you specify `.only` or
+    `.without` as it could potentially cause data loss.
+- `per_document` task `scope` now correctly works with `.includes` even
+    in parallel execution mode.
+
 ## v2.0.0
 
-- `Hekenga::Iterator` has been replaced by `Hekenga::IdIterator`. If any
+- (breaking) `Hekenga::Iterator` has been replaced by `Hekenga::IdIterator`. If any
     selector or sort is set on a document task migration scope, it no longer forces an
     ascending ID sort. This should help to prevent index misses, though there is a
     tradeoff that documents being concurrently updated may be skipped or

data/CLAUDE.md

@@ -0,0 +1,60 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Hekenga is a Ruby gem providing a migration framework for MongoDB (via Mongoid). It supports sequential and parallel document processing via ActiveJob, with error recovery, validation tracking, and a Thor-based CLI.
+
+## Common Commands
+
+```bash
+# Run full test suite (requires MongoDB - see docker-compose.yml)
+rake spec
+
+# Run a single spec file
+rake spec SPEC=spec/hekenga/document_task_spec.rb
+
+# Install gem locally
+bundle exec rake install
+
+# Interactive console with gem loaded
+bin/console
+```
+
+## Architecture
+
+### Migration Flow
+
+```
+Migration.perform! → MasterProcess.run! → launches tasks in threads
+  SimpleTask:   executes up/down blocks directly
+  DocumentTask: iterates documents → batch → execute → write (sequential)
+  ParallelTask: splits into ID batches → enqueues ParallelJob per batch (via ActiveJob)
+```
+
+### Key Components
+
+- **`Hekenga::Migration`** — main migration class, orchestrates tasks
+- **`Hekenga::MasterProcess`** — launches tasks, manages execution/recovery/progress
+- **`Hekenga::DSL::*`** — fluent DSL for defining migrations (`DSL::Migration`, `DSL::SimpleTask`, `DSL::DocumentTask`)
+- **`Hekenga::DocumentTaskExecutor`** — core document processing: filter → up block → validate → write
+- **`Hekenga::ParallelTask`** / **`Hekenga::ParallelJob`** — parallel execution via ActiveJob
+- **`Hekenga::DocumentTaskRecord`** — Mongoid doc tracking parallel task progress
+- **`Hekenga::Log`** — Mongoid doc tracking migration/task status (`:naught`, `:running`, `:complete`, `:failed`, `:skipped`)
+- **`Hekenga::Failure::*`** — error/validation/write/cancelled failure tracking subclasses
+- **`Hekenga::IdIterator`** / **`Hekenga::MongoidIterator`** — efficient document iteration for parallel vs sequential paths
+
+### Task Types
+
+- **SimpleTask** — one-off up/down blocks, no document iteration
+- **DocumentTask** — per-document processing with scope, filter, setup, up, down, after blocks; supports `parallel!`, `timeless!`, `always_write!`, `use_transaction!`, configurable write strategies (`:update` vs `:delete_then_insert`)
+
+### Configuration
+
+Via `Hekenga.configure` block — sets migration directory and report frequency. Thread-safe registry tracks all migrations.
+
+## Dependencies
+
+- **mongoid** (>= 6), **activejob** (>= 5), **thor** (1.2.1)
+- Test: **rspec** (~> 3.0), **database_cleaner-mongoid** (~> 2.0), **pry**

data/README.md

@@ -1,7 +1,7 @@
 # Hekenga
 
-An attempt at a migration framework for MongoDB that supports parallel document
-processing via ActiveJob, chained jobs and error recovery.
+A migration framework for MongoDB (via Mongoid) that supports parallel document
+processing via ActiveJob, chained jobs, and error recovery.
 
 ## Installation
 
@@ -19,13 +19,151 @@ Or install it yourself as:
 
     $ gem install hekenga
 
+## Configuration
+
+```ruby
+Hekenga.configure do |config|
+  config.dir  = ["db", "hekenga"] # where migration files live (relative to root)
+  config.root = Dir.pwd           # application root
+end
+```
+
+Migrations are stored as Ruby files in the configured directory (default: `db/hekenga/`).
+
 ## Usage
 
-CLI instructions:
+### CLI
+
+```
+$ hekenga help                          # Show all available commands
+$ hekenga generate <description>        # Generate a new migration scaffold
+$ hekenga status                        # Show status of all migrations
+$ hekenga run_all!                      # Run all pending migrations in date order
+$ hekenga run! <path_or_pkey>           # Run a specific migration
+$ hekenga run! <path_or_pkey> --test    # Dry run (no writes persisted)
+$ hekenga run! <path_or_pkey> --clear   # Clear logs before running
+$ hekenga recover! <path_or_pkey>       # Re-process failed/invalid records
+$ hekenga watch <path_or_pkey>          # Attach to a running migration, report status periodically
+$ hekenga failures <path_or_pkey>       # Print all failed/invalid document IDs for a migration
+$ hekenga cancel                        # Cancel all active migrations
+$ hekenga skip <path_or_pkey>           # Mark a migration as skipped
+$ hekenga clear! <path_or_pkey>         # Remove all logs/failures for a migration
+$ hekenga cleanup                       # Remove all failure logs
+```
+
+### Writing Migrations
+
+Generate a migration scaffold:
+
+    $ hekenga generate "Add default role to users"
+
+#### Simple Tasks
+
+Simple tasks run arbitrary code once. Use `actual?` and `test?` to check execution mode.
+
+```ruby
+Hekenga.migration do
+  description "Backfill analytics collection"
+  created "2024-01-15 10:00"
+
+  task "Create indexes" do
+    up do
+      Analytics.create_indexes if actual?
+    end
+  end
+end
+```
+
+#### Document Tasks
+
+Document tasks iterate over a Mongoid scope and process each document in batches.
+
+```ruby
+Hekenga.migration do
+  description "Normalize user emails"
+  created "2024-01-15 10:00"
+  batch_size 100 # default batch size for all tasks in this migration
+
+  per_document "Downcase emails" do
+    scope User.all
+
+    # Called once per batch; instance variables are shared with filter/up/after
+    setup do |docs|
+      @domain_map = ExternalService.load_domains
+    end
+
+    # Return false to skip a document
+    filter do |doc|
+      doc.email.present?
+    end
+
+    # Mutate the document in place — Hekenga handles persistence
+    up do |doc|
+      doc.email = doc.email.downcase
+    end
+
+    # Called once per batch with the successfully written documents
+    after do |docs|
+      AuditLog.record(docs.map(&:id))
+    end
+  end
+end
+```
+
+#### Document Task Options
+
+```ruby
+per_document "Process records" do
+  scope MyModel.where(active: true)
+
+  parallel!                           # Process batches in parallel via ActiveJob
+  timeless!                           # Don't update Mongoid timestamps
+  always_write!                       # Write even if the document didn't change
+  skip_prepare!                       # Skip Mongoid callbacks on load
+  skip_validation!                    # Write documents without running Mongoid validations
+  use_transaction!                    # Wrap each batch in a MongoDB transaction
+  batch_size 50                       # Override migration-level batch size
+  write_strategy :update              # :update (default) or :delete_then_insert
+  cursor_timeout 86_400               # Max cursor lifetime in seconds (default: 1 day)
+
+  up do |doc|
+    doc.status = "migrated"
+  end
+end
+```
+
+### Test Mode
+
+Run a migration without persisting changes:
+
+```ruby
+migration = Hekenga.find_migration("2024-01-15-add-default-role-to-users")
+migration.test_mode!
+migration.perform!
+```
+
+Or via the CLI:
+
+    $ hekenga run! <path_or_pkey> --test
+
+### Monitoring
+
+Attach to a migration that's already running (for example one launched in another process or via a background job) and print its status on a fixed interval:
+
+    $ hekenga watch <path_or_pkey>
+    $ hekenga watch <path_or_pkey> --interval 5   # report every 5 seconds
+
+The reporting interval defaults to `Hekenga.config.report_sleep`.
+
+### Recovery
+
+When a migration fails (due to errors, invalid records, or write failures), Hekenga logs the failures and marks the migration as failed. You can re-process only the failed records:
+
+    $ hekenga recover! <path_or_pkey>
 
-    $ hekenga help
+To inspect exactly which documents failed or were invalid across all document tasks in a migration:
 
-Migration DSL documentation TBD, for now please look at spec/
+    $ hekenga failures <path_or_pkey>
 
 ## Development
 

data/docker-compose.yml

@@ -9,22 +9,41 @@ networks:
 services:
   mongo:
     image: mongo:6
-    command: ["--replSet", "rs0", "--bind_ip", "localhost,mongo"]
+    command: ["--replSet", "rs0", "--bind_ip_all"]
     volumes:
       - mongo:/data/db
     ports:
       - 27017:27017
     networks:
       - hekenga-net
+    healthcheck:
+      # mongo:6 ships mongosh (the legacy `mongo` shell was removed in 6.0).
+      test: ["CMD", "mongosh", "--quiet", "--eval", "db.adminCommand('ping')"]
+      interval: 5s
+      timeout: 5s
+      retries: 12
 
   mongosetup:
     image: mongo:6
     depends_on:
-      - mongo
+      mongo:
+        condition: service_healthy
     restart: "no"
     entrypoint:
-      - bash
-      - "-c"
-      - "sleep 3 && mongo --host mongo:27017 --eval 'rs.initiate({_id: \"rs0\", members: [{_id: 0, host: \"localhost:27017\"}]})'"
+      - mongosh
+      - "--quiet"
+      - "--host"
+      - "mongo:27017"
+      - "--eval"
+      # Idempotent: initiate the single-member replica set only if it hasn't
+      # been configured yet, so re-running compose doesn't error. The member
+      # advertises as localhost:27017 so the driver on the host can reach it
+      # after replica-set topology discovery (required for transactions).
+      - >
+        try { rs.status(); print('replica set already initialised'); }
+        catch (e) {
+          rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'localhost:27017'}]});
+          print('replica set initiated');
+        }
     networks:
       - hekenga-net

data/exe/hekenga

@@ -71,6 +71,20 @@ class HekengaCLI < Thor
     end
   end
 
+  desc "watch PATH_OR_PKEY", "Attach to a running migration and report its status periodically."
+  option :interval, type: :numeric, desc: "Seconds between status reports (default: Hekenga.config.report_sleep)."
+  def watch(path_or_pkey)
+    migration = load_migration(path_or_pkey)
+    interval  = options[:interval] || Hekenga.config.report_sleep
+    Hekenga::MasterProcess.new(migration).watch!(interval: interval)
+  end
+
+  desc "failures PATH_OR_PKEY", "Print all failed and invalid document IDs for a migration."
+  def failures(path_or_pkey)
+    migration = load_migration(path_or_pkey)
+    Hekenga::FailureReport.new(migration).print!
+  end
+
   desc "cancel", "Cancel all active migrations."
   def cancel
     Hekenga::Log.where(done: false).set(cancel: true)

data/lib/hekenga/document_task.rb

@@ -5,7 +5,7 @@ module Hekenga
     attr_reader :ups, :downs, :setups, :filters, :after_callbacks
     attr_accessor :parallel, :scope, :timeless, :batch_size, :cursor_timeout
     attr_accessor :description, :invalid_strategy, :skip_prepare, :write_strategy
-    attr_accessor :always_write, :use_transaction
+    attr_accessor :always_write, :use_transaction, :skip_validation
 
     def initialize
       @ups              = []
@@ -16,6 +16,7 @@ module Hekenga
       @invalid_strategy = :continue
       @write_strategy   = :update
       @skip_prepare     = false
+      @skip_validation  = false
       @batch_size       = nil
       @always_write     = false
       @use_transaction  = false
@@ -24,6 +25,9 @@ module Hekenga
 
     def validate!
       raise Hekenga::Invalid.new(self, :ups, "missing") unless ups.any?
+      if scope&.options&.key?(:fields)
+        raise Hekenga::Invalid.new(self, :scope, "uses .only() or .without() which would cause data loss with replace_one")
+      end
     end
 
     def up!(context, document)

data/lib/hekenga/document_task_executor.rb

@@ -59,7 +59,11 @@ module Hekenga
     end
 
     def record_scope
-      task.scope.klass.unscoped.in(_id: task_record.ids)
+      scope = task.scope.klass.unscoped.in(_id: task_record.ids)
+      if task.scope.inclusions.any?
+        scope = scope.includes(*task.scope.inclusions.map(&:name))
+      end
+      scope
     end
 
     def records
@@ -112,6 +116,8 @@ module Hekenga
     end
 
     def validate_records
+      return records_to_write.concat(migrated_records) if task.skip_validation
+
       migrated_records.each do |record|
         if record.valid?
           records_to_write << record

data/lib/hekenga/dsl/document_task.rb

@@ -56,6 +56,10 @@ module Hekenga
         @object.skip_prepare = true
       end
 
+      def skip_validation!
+        @object.skip_validation = true
+      end
+
       def setup(&block)
         @object.setups.push block
       end

data/lib/hekenga/failure_report.rb

@@ -0,0 +1,60 @@
+module Hekenga
+  class FailureReport
+    def initialize(migration)
+      @migration = migration
+    end
+
+    # Per-task summaries for document tasks that have any failed or invalid
+    # document IDs.
+    # => [{ idx:, description:, failed_ids: [...], invalid_ids: [...] }, ...]
+    def tasks
+      @migration.tasks.each.with_index.filter_map do |task, idx|
+        next unless task.is_a?(Hekenga::DocumentTask)
+
+        records = @migration.task_records(idx).any_of(
+          { :failed_ids.ne  => [] },
+          { :invalid_ids.ne => [] }
+        )
+
+        failed_ids  = []
+        invalid_ids = []
+        records.pluck(:failed_ids, :invalid_ids).each do |failed, invalid|
+          failed_ids.concat(failed)   if failed
+          invalid_ids.concat(invalid) if invalid
+        end
+        next if failed_ids.empty? && invalid_ids.empty?
+
+        {
+          idx:         idx,
+          description: task.description,
+          failed_ids:  failed_ids,
+          invalid_ids: invalid_ids
+        }
+      end
+    end
+
+    def print!
+      Hekenga.log("Failures for #{@migration.to_key}")
+
+      summaries = tasks
+      if summaries.empty?
+        Hekenga.log("  No failed or invalid documents.")
+        return
+      end
+
+      failed_ids  = summaries.flat_map { |summary| summary[:failed_ids] }
+      invalid_ids = summaries.flat_map { |summary| summary[:invalid_ids] }
+
+      print_ids("Failed",  failed_ids)
+      print_ids("Invalid", invalid_ids)
+    end
+
+    private
+
+    def print_ids(label, ids)
+      return if ids.empty?
+
+      Hekenga.log("    #{label} (#{ids.length}): #{ids.map { |id| "\"#{id}\"" }.join(", ")}")
+    end
+  end
+end

data/lib/hekenga/master_process.rb

@@ -33,18 +33,35 @@ module Hekenga
       true
     end
 
+    def watch!(interval: Hekenga.config.report_sleep)
+      if Hekenga.status(@migration) == :naught
+        Hekenga.log "Migration #{@migration.to_key} has not started yet."
+        return false
+      end
+      Hekenga.log "Watching migration #{@migration.to_key}: #{@migration.description}"
+      @migration.tasks.each.with_index do |task, idx|
+        report_while_active(task, idx, interval: interval)
+      rescue Hekenga::TaskFailedError
+        return false
+      end
+      true
+    end
+
     private
 
-    def report_while_active(task, idx)
-      # Wait for the log to be generated
+    def report_while_active(task, idx, interval: Hekenga.config.report_sleep)
+      # Wait for the log to be generated. When watching an out-of-process
+      # migration there is no @active_thread guaranteeing it, so bail out if
+      # the migration finished without ever reaching this task.
       until (@migration.log(idx) rescue nil)
+        return if %i[complete skipped].include?(Hekenga.status(@migration))
         sleep 1
       end
-      # Periodically report on thread progress
+      # Periodically report on progress
       until @migration.log(idx).reload.done
-        @active_thread.join
+        @active_thread&.join
         report_status(task, idx)
-        sleep Hekenga.config.report_sleep
+        sleep interval
       end
       report_status(task, idx) if task.is_a?(Hekenga::DocumentTask)
       report_result(task, idx)

data/lib/hekenga/version.rb

@@ -1,3 +1,3 @@
 module Hekenga
-  VERSION = "2.0.0"
+  VERSION = "2.2.0"
 end

data/lib/hekenga.rb

@@ -6,6 +6,7 @@ require "hekenga/config"
 require "hekenga/irreversible"
 require "hekenga/virtual_method"
 require "hekenga/scaffold"
+require "hekenga/failure_report"
 
 module Hekenga
   @@load_all_mutex = Mutex.new

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hekenga
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.2.0
 platform: ruby
 authors:
 - Tapio Saarinen
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-07-31 00:00:00.000000000 Z
+date: 2026-07-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -148,6 +148,7 @@ files:
 - ".rspec"
 - ".travis.yml"
 - CHANGELOG.md
+- CLAUDE.md
 - Gemfile
 - README.md
 - Rakefile
@@ -174,6 +175,7 @@ files:
 - lib/hekenga/failure/error.rb
 - lib/hekenga/failure/validation.rb
 - lib/hekenga/failure/write.rb
+- lib/hekenga/failure_report.rb
 - lib/hekenga/id_iterator.rb
 - lib/hekenga/invalid.rb
 - lib/hekenga/irreversible.rb