hekenga 1.1.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 94824a7ff42ed31e87c54b5093ad9b8a03b27b08ec5cc7c4f45491b71b3dcc79
-  data.tar.gz: ca0ffaded81560273fa6421de3dfdb9a0efce0fd62f95c410bd31004cb17a816
+  metadata.gz: d5a372f23cc2fe1751eb08e07e40705b0361fc609b072e14226b72819cfe9647
+  data.tar.gz: 99228c0f076660abe23c92f37c55509b156810fc0a741764b7872d4a0c597263
 SHA512:
-  metadata.gz: 8677a4cd8023f511971b54ac4458e7b230ee693246473d77b9e3237895ef95670aa157d0117f13223a0df4dde71430de9c940bfc89315422203e23ec3434218e
-  data.tar.gz: 69b879e26c4d7771377881d9e83d411fa3b0a2674ce851b5b9b74f323c1c2dec749218257766b27e467eb409708f221d9125a8451bd9d394b103e654e86785da
+  metadata.gz: e2dfea01ce0c1c17fc51ab550e1805653a255d7c65a8b8424c7a1d6e646d59b9bc70fc45cfcf558d1fd7a6df81e5a2a8230a240e8c75d578736cdf19f4e2e290
+  data.tar.gz: 0af8e27a9f1b4c7ae490788f64457a67fae9a3fb9459821b8066bca61671ae30463971e9ec277d01ba358817992260281e5dceaaab676f9093dc05a941a520ce

data/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## v2.1.0
+
+- `per_document` task `scope` will no longer let you specify `.only` or
+    `.without` as it could potentially cause data loss.
+- `per_document` task `scope` now correctly works with `.includes` even
+    in parallel execution mode.
+
+## v2.0.0
+
+- (breaking) `Hekenga::Iterator` has been replaced by `Hekenga::IdIterator`. If any
+    selector or sort is set on a document task migration scope, it no longer forces an
+    ascending ID sort. This should help to prevent index misses, though there is a
+    tradeoff that documents being concurrently updated may be skipped or
+    processed multiple times. Hekenga tries to guard against processing multiple
+    times. Manually specifying an `asc(:_id)` on your scope will continue to
+    process documents in ID order.
+- Document tasks now support a new option, `cursor_timeout`. This is the maximum
+    time a document task's `scope` can be iterated and queue jobs within. The
+    default is one day.
+
 ## v1.1.0
 
 - `setup` is now passed the current batch of documents so it can be used to

data/CLAUDE.md

@@ -0,0 +1,60 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Hekenga is a Ruby gem providing a migration framework for MongoDB (via Mongoid). It supports sequential and parallel document processing via ActiveJob, with error recovery, validation tracking, and a Thor-based CLI.
+
+## Common Commands
+
+```bash
+# Run full test suite (requires MongoDB - see docker-compose.yml)
+rake spec
+
+# Run a single spec file
+rake spec SPEC=spec/hekenga/document_task_spec.rb
+
+# Install gem locally
+bundle exec rake install
+
+# Interactive console with gem loaded
+bin/console
+```
+
+## Architecture
+
+### Migration Flow
+
+```
+Migration.perform! → MasterProcess.run! → launches tasks in threads
+  SimpleTask:   executes up/down blocks directly
+  DocumentTask: iterates documents → batch → execute → write (sequential)
+  ParallelTask: splits into ID batches → enqueues ParallelJob per batch (via ActiveJob)
+```
+
+### Key Components
+
+- **`Hekenga::Migration`** — main migration class, orchestrates tasks
+- **`Hekenga::MasterProcess`** — launches tasks, manages execution/recovery/progress
+- **`Hekenga::DSL::*`** — fluent DSL for defining migrations (`DSL::Migration`, `DSL::SimpleTask`, `DSL::DocumentTask`)
+- **`Hekenga::DocumentTaskExecutor`** — core document processing: filter → up block → validate → write
+- **`Hekenga::ParallelTask`** / **`Hekenga::ParallelJob`** — parallel execution via ActiveJob
+- **`Hekenga::DocumentTaskRecord`** — Mongoid doc tracking parallel task progress
+- **`Hekenga::Log`** — Mongoid doc tracking migration/task status (`:naught`, `:running`, `:complete`, `:failed`, `:skipped`)
+- **`Hekenga::Failure::*`** — error/validation/write/cancelled failure tracking subclasses
+- **`Hekenga::IdIterator`** / **`Hekenga::MongoidIterator`** — efficient document iteration for parallel vs sequential paths
+
+### Task Types
+
+- **SimpleTask** — one-off up/down blocks, no document iteration
+- **DocumentTask** — per-document processing with scope, filter, setup, up, down, after blocks; supports `parallel!`, `timeless!`, `always_write!`, `use_transaction!`, configurable write strategies (`:update` vs `:delete_then_insert`)
+
+### Configuration
+
+Via `Hekenga.configure` block — sets migration directory and report frequency. Thread-safe registry tracks all migrations.
+
+## Dependencies
+
+- **mongoid** (>= 6), **activejob** (>= 5), **thor** (1.2.1)
+- Test: **rspec** (~> 3.0), **database_cleaner-mongoid** (~> 2.0), **pry**

data/README.md

@@ -1,10 +1,7 @@
 # Hekenga
 
-An attempt at a migration framework for MongoDB that supports parallel document
-processing via ActiveJob, chained jobs and error recovery.
-
-**Note that this gem is currently in pre-alpha - assume most things have a high
-chance of being broken.**
+A migration framework for MongoDB (via Mongoid) that supports parallel document
+processing via ActiveJob, chained jobs, and error recovery.
 
 ## Installation
 
@@ -22,13 +19,135 @@ Or install it yourself as:
 
     $ gem install hekenga
 
+## Configuration
+
+```ruby
+Hekenga.configure do |config|
+  config.dir  = ["db", "hekenga"] # where migration files live (relative to root)
+  config.root = Dir.pwd           # application root
+end
+```
+
+Migrations are stored as Ruby files in the configured directory (default: `db/hekenga/`).
+
 ## Usage
 
-CLI instructions:
+### CLI
+
+```
+$ hekenga help                          # Show all available commands
+$ hekenga generate <description>        # Generate a new migration scaffold
+$ hekenga status                        # Show status of all migrations
+$ hekenga run_all!                      # Run all pending migrations in date order
+$ hekenga run! <path_or_pkey>           # Run a specific migration
+$ hekenga run! <path_or_pkey> --test    # Dry run (no writes persisted)
+$ hekenga run! <path_or_pkey> --clear   # Clear logs before running
+$ hekenga recover! <path_or_pkey>       # Re-process failed/invalid records
+$ hekenga cancel                        # Cancel all active migrations
+$ hekenga skip <path_or_pkey>           # Mark a migration as skipped
+$ hekenga clear! <path_or_pkey>         # Remove all logs/failures for a migration
+$ hekenga cleanup                       # Remove all failure logs
+```
+
+### Writing Migrations
+
+Generate a migration scaffold:
+
+    $ hekenga generate "Add default role to users"
+
+#### Simple Tasks
+
+Simple tasks run arbitrary code once. Use `actual?` and `test?` to check execution mode.
+
+```ruby
+Hekenga.migration do
+  description "Backfill analytics collection"
+  created "2024-01-15 10:00"
+
+  task "Create indexes" do
+    up do
+      Analytics.create_indexes if actual?
+    end
+  end
+end
+```
+
+#### Document Tasks
+
+Document tasks iterate over a Mongoid scope and process each document in batches.
+
+```ruby
+Hekenga.migration do
+  description "Normalize user emails"
+  created "2024-01-15 10:00"
+  batch_size 100 # default batch size for all tasks in this migration
+
+  per_document "Downcase emails" do
+    scope User.all
+
+    # Called once per batch; instance variables are shared with filter/up/after
+    setup do |docs|
+      @domain_map = ExternalService.load_domains
+    end
+
+    # Return false to skip a document
+    filter do |doc|
+      doc.email.present?
+    end
+
+    # Mutate the document in place — Hekenga handles persistence
+    up do |doc|
+      doc.email = doc.email.downcase
+    end
+
+    # Called once per batch with the successfully written documents
+    after do |docs|
+      AuditLog.record(docs.map(&:id))
+    end
+  end
+end
+```
+
+#### Document Task Options
+
+```ruby
+per_document "Process records" do
+  scope MyModel.where(active: true)
+
+  parallel!                           # Process batches in parallel via ActiveJob
+  timeless!                           # Don't update Mongoid timestamps
+  always_write!                       # Write even if the document didn't change
+  skip_prepare!                       # Skip Mongoid callbacks on load
+  use_transaction!                    # Wrap each batch in a MongoDB transaction
+  batch_size 50                       # Override migration-level batch size
+  write_strategy :update              # :update (default) or :delete_then_insert
+  cursor_timeout 86_400               # Max cursor lifetime in seconds (default: 1 day)
+
+  up do |doc|
+    doc.status = "migrated"
+  end
+end
+```
+
+### Test Mode
+
+Run a migration without persisting changes:
+
+```ruby
+migration = Hekenga.find_migration("2024-01-15-add-default-role-to-users")
+migration.test_mode!
+migration.perform!
+```
+
+Or via the CLI:
+
+    $ hekenga run! <path_or_pkey> --test
+
+### Recovery
 
-    $ hekenga help
+When a migration fails (due to errors, invalid records, or write failures), Hekenga logs the failures and marks the migration as failed. You can re-process only the failed records:
 
-Migration DSL documentation TBD, for now please look at spec/
+    $ hekenga recover! <path_or_pkey>
 
 ## Development
 

data/docker-compose.yml

@@ -8,7 +8,7 @@ networks:
 
 services:
   mongo:
-    image: mongo:5
+    image: mongo:6
     command: ["--replSet", "rs0", "--bind_ip", "localhost,mongo"]
     volumes:
       - mongo:/data/db
@@ -18,7 +18,7 @@ services:
       - hekenga-net
 
   mongosetup:
-    image: mongo:5
+    image: mongo:6
     depends_on:
       - mongo
     restart: "no"

data/lib/hekenga/base_iterator.rb

@@ -0,0 +1,24 @@
+module Hekenga
+  class BaseIterator
+    include Enumerable
+    DEFAULT_TIMEOUT = 86_400 # 1 day in seconds
+
+    attr_reader :cursor_timeout
+
+    def initialize(scope:, cursor_timeout: DEFAULT_TIMEOUT)
+      @scope = scope
+      @cursor_timeout = cursor_timeout
+    end
+
+    private
+
+    def iteration_scope
+      if @scope.selector.blank? && @scope.options.blank?
+        # Apply a default _id sort, it works the best
+        @scope.asc(:_id)
+      else
+        @scope
+      end.max_time_ms(cursor_timeout * 1000) # convert to ms
+    end
+  end
+end

data/lib/hekenga/document_task.rb

@@ -1,8 +1,9 @@
 require 'hekenga/irreversible'
+require 'hekenga/base_iterator'
 module Hekenga
   class DocumentTask
     attr_reader :ups, :downs, :setups, :filters, :after_callbacks
-    attr_accessor :parallel, :scope, :timeless, :batch_size
+    attr_accessor :parallel, :scope, :timeless, :batch_size, :cursor_timeout
     attr_accessor :description, :invalid_strategy, :skip_prepare, :write_strategy
     attr_accessor :always_write, :use_transaction
 
@@ -18,10 +19,14 @@ module Hekenga
       @batch_size       = nil
       @always_write     = false
       @use_transaction  = false
+      @cursor_timeout   = Hekenga::BaseIterator::DEFAULT_TIMEOUT
     end
 
     def validate!
       raise Hekenga::Invalid.new(self, :ups, "missing") unless ups.any?
+      if scope&.options&.key?(:fields)
+        raise Hekenga::Invalid.new(self, :scope, "uses .only() or .without() which would cause data loss with replace_one")
+      end
     end
 
     def up!(context, document)

data/lib/hekenga/document_task_executor.rb

@@ -59,7 +59,11 @@ module Hekenga
     end
 
     def record_scope
-      task.scope.klass.unscoped.in(_id: task_record.ids)
+      scope = task.scope.klass.unscoped.in(_id: task_record.ids)
+      if task.scope.inclusions.any?
+        scope = scope.includes(*task.scope.inclusions.map(&:name))
+      end
+      scope
     end
 
     def records

data/lib/hekenga/dsl/document_task.rb

@@ -28,6 +28,10 @@ module Hekenga
         @object.write_strategy = strategy
       end
 
+      def cursor_timeout(timeout)
+        @object.cursor_timeout = timeout.to_i
+      end
+
       def scope(scope)
         @object.scope = scope
       end

data/lib/hekenga/id_iterator.rb

@@ -0,0 +1,34 @@
+require "hekenga/base_iterator"
+module Hekenga
+  class IdIterator < BaseIterator
+    DEFAULT_ID = "_id".freeze
+
+    attr_reader :id_property
+
+    def initialize(id_property: DEFAULT_ID, **kwargs)
+      super(**kwargs)
+      @id_property = id_property
+    end
+
+    def each
+      with_view do |view|
+        view.each do |doc|
+          yield doc[id_property]
+        end
+      end
+    end
+
+    private
+
+    def with_view
+      view = iteration_scope.view
+      yield view
+    ensure
+      view.close_query
+    end
+
+    def iteration_scope
+      super.only(id_property)
+    end
+  end
+end

data/lib/hekenga/migration.rb

@@ -2,6 +2,7 @@ require 'hekenga/invalid'
 require 'hekenga/context'
 require 'hekenga/parallel_job'
 require 'hekenga/parallel_task'
+require 'hekenga/mongoid_iterator'
 require 'hekenga/master_process'
 require 'hekenga/document_task_record'
 require 'hekenga/document_task_executor'
@@ -132,18 +133,18 @@ module Hekenga
       records = []
       task_records(task_idx).delete_all unless recover
       executor_key = BSON::ObjectId.new
-      task.scope.asc(:_id).no_timeout.each do |record|
+      Hekenga::MongoidIterator.new(scope: task.scope, cursor_timeout: task.cursor_timeout).each do |record|
         records.push(record)
         next unless records.length == (task.batch_size || batch_size)
 
-        records = filter_out_processed(task, task_idx, records) if recover
+        records = filter_out_processed(task, task_idx, records)
         next unless records.length == (task.batch_size || batch_size)
 
         execute_document_task(task_idx, executor_key, records)
         records = []
         return if log.cancel
       end
-      records = filter_out_processed(task, task_idx, records) if recover
+      records = filter_out_processed(task, task_idx, records)
       execute_document_task(task_idx, executor_key, records) if records.any?
       return if log.cancel
       log_done!

data/lib/hekenga/mongoid_iterator.rb

@@ -0,0 +1,8 @@
+require "hekenga/base_iterator"
+module Hekenga
+  class MongoidIterator < BaseIterator
+    def each(&block)
+      iteration_scope.each(&block)
+    end
+  end
+end

data/lib/hekenga/parallel_task.rb

@@ -1,4 +1,4 @@
-require 'hekenga/iterator'
+require 'hekenga/id_iterator'
 require 'hekenga/document_task_executor'
 require 'hekenga/task_splitter'
 
@@ -15,13 +15,13 @@ module Hekenga
 
     def start!
       clear_task_records!
-      @executor_key = BSON::ObjectId.new
+      regenerate_executor_key
       generate_for_scope(task.scope)
       check_for_completion!
     end
 
     def resume!
-      @executor_key = BSON::ObjectId.new
+      regenerate_executor_key
       task_records.set(executor_key: @executor_key)
       queue_jobs!(task_records.incomplete)
       generate_new_records!
@@ -41,16 +41,43 @@ module Hekenga
 
     private
 
+    def regenerate_executor_key
+      @executor_key = BSON::ObjectId.new
+    end
+
     def generate_for_scope(scope)
-      Hekenga::Iterator.new(scope, size: 100_000).each do |id_block|
-        task_records = id_block.each_slice(batch_size).map do |id_slice|
-          generate_task_records!(id_slice)
-        end
+      Hekenga::IdIterator.new(
+        scope: scope,
+        cursor_timeout: task.cursor_timeout
+        # Batch Batches of IDs
+      ).each_slice(batch_size).each_slice(enqueue_size) do |id_block|
+        sanitize_id_block!(id_block)
+        task_records = id_block.reject(&:empty?).map(&method(:generate_task_record!))
         write_task_records!(task_records)
         queue_jobs!(task_records)
       end
     end
 
+    def enqueue_size
+      500 # task records written + enqueued at a time
+    end
+
+    def sanitize_id_block!(id_block)
+      return if task.scope.options.blank? && task.scope.selector.blank?
+
+      # Custom ordering on cursor with parallel updates may result in the same
+      # ID getting yielded into the migration multiple times. Detect this +
+      # remove
+      doubleups = task_records.in(ids: id_block.flatten).pluck(:ids).flatten.to_set
+      return if doubleups.empty?
+
+      id_block.each do |id_slice|
+        id_slice.reject! do |id|
+          doubleups.include?(id)
+        end
+      end
+    end
+
     def generate_new_records!
       last_record = task_records.desc(:_id).first
       last_id = last_record&.ids&.last
@@ -83,7 +110,7 @@ module Hekenga
       migration.task_records(task_idx)
     end
 
-    def generate_task_records!(id_slice)
+    def generate_task_record!(id_slice)
       Hekenga::DocumentTaskRecord.new(
         migration_key: migration.to_key,
         task_idx:      task_idx,

data/lib/hekenga/scaffold.rb

@@ -48,6 +48,7 @@ module Hekenga
         #  #skip_prepare!
         #  #batch_size 25
         #  #write_strategy :update # :delete_then_insert
+        #  #cursor_timeout 86_400 # max allowed time for the cursor to survive, in seconds
         #
         #  # Called once per batch, instance variables will be accessible
         #  # in the filter, up and after blocks

data/lib/hekenga/version.rb

@@ -1,3 +1,3 @@
 module Hekenga
-  VERSION = "1.1.0"
+  VERSION = "2.1.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hekenga
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 2.1.0
 platform: ruby
 authors:
 - Tapio Saarinen
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-02-20 00:00:00.000000000 Z
+date: 2026-04-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -148,6 +148,7 @@ files:
 - ".rspec"
 - ".travis.yml"
 - CHANGELOG.md
+- CLAUDE.md
 - Gemfile
 - README.md
 - Rakefile
@@ -159,6 +160,7 @@ files:
 - hekenga.gemspec
 - lib/hekenga.rb
 - lib/hekenga/base_error.rb
+- lib/hekenga/base_iterator.rb
 - lib/hekenga/config.rb
 - lib/hekenga/context.rb
 - lib/hekenga/document_task.rb
@@ -173,12 +175,13 @@ files:
 - lib/hekenga/failure/error.rb
 - lib/hekenga/failure/validation.rb
 - lib/hekenga/failure/write.rb
+- lib/hekenga/id_iterator.rb
 - lib/hekenga/invalid.rb
 - lib/hekenga/irreversible.rb
-- lib/hekenga/iterator.rb
 - lib/hekenga/log.rb
 - lib/hekenga/master_process.rb
 - lib/hekenga/migration.rb
+- lib/hekenga/mongoid_iterator.rb
 - lib/hekenga/parallel_job.rb
 - lib/hekenga/parallel_task.rb
 - lib/hekenga/scaffold.rb

data/lib/hekenga/iterator.rb

@@ -1,26 +0,0 @@
-module Hekenga
-  class Iterator
-    include Enumerable
-
-    SMALLEST_ID = BSON::ObjectId.from_string('0'*24)
-
-    attr_reader :scope, :size
-
-    def initialize(scope, size:)
-      @scope = scope
-      @size = size
-    end
-
-    def each(&block)
-      current_id = SMALLEST_ID
-      base_scope = scope.asc(:_id).limit(size)
-
-      loop do
-        ids = base_scope.and(_id: {'$gt': current_id}).pluck(:_id)
-        break if ids.empty?
-        yield ids
-        current_id = ids.sort.last
-      end
-    end
-  end
-end