hekenga 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f21c3c1cb0e45c3b9eb2627fc960d032ff981e01b239dc69c02d2aebc1f7b539
-  data.tar.gz: 62fbbd8a65bae8bacc537bcc40cc4a3960b795ebb19a13c34b8ebd539eb102d0
+  metadata.gz: d5a372f23cc2fe1751eb08e07e40705b0361fc609b072e14226b72819cfe9647
+  data.tar.gz: 99228c0f076660abe23c92f37c55509b156810fc0a741764b7872d4a0c597263
 SHA512:
-  metadata.gz: 6317b298a05085564cfaaee1beef6b5749e83f8bc877538b448db66b50c3dde50f0380a5fa7f178dff7feca6840564c486f263891a838d4ed51617b7ff4f8698
-  data.tar.gz: 12ba1c564acca2f7d43c80a384a0767b00592abf5126d0016a0bf7cadf610e6d342890cd54597a602654e8449491135a3dda488bb7c1c8aaeb72d035bdc7c5d6
+  metadata.gz: e2dfea01ce0c1c17fc51ab550e1805653a255d7c65a8b8424c7a1d6e646d59b9bc70fc45cfcf558d1fd7a6df81e5a2a8230a240e8c75d578736cdf19f4e2e290
+  data.tar.gz: 0af8e27a9f1b4c7ae490788f64457a67fae9a3fb9459821b8066bca61671ae30463971e9ec277d01ba358817992260281e5dceaaab676f9093dc05a941a520ce

data/CHANGELOG.md

@@ -1,8 +1,15 @@
 # Changelog
 
+## v2.1.0
+
+- `per_document` task `scope` will no longer let you specify `.only` or
+    `.without` as it could potentially cause data loss.
+- `per_document` task `scope` now correctly works with `.includes` even
+    in parallel execution mode.
+
 ## v2.0.0
 
-- `Hekenga::Iterator` has been replaced by `Hekenga::IdIterator`. If any
+- (breaking) `Hekenga::Iterator` has been replaced by `Hekenga::IdIterator`. If any
     selector or sort is set on a document task migration scope, it no longer forces an
     ascending ID sort. This should help to prevent index misses, though there is a
     tradeoff that documents being concurrently updated may be skipped or

data/CLAUDE.md

@@ -0,0 +1,60 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Hekenga is a Ruby gem providing a migration framework for MongoDB (via Mongoid). It supports sequential and parallel document processing via ActiveJob, with error recovery, validation tracking, and a Thor-based CLI.
+
+## Common Commands
+
+```bash
+# Run full test suite (requires MongoDB - see docker-compose.yml)
+rake spec
+
+# Run a single spec file
+rake spec SPEC=spec/hekenga/document_task_spec.rb
+
+# Install gem locally
+bundle exec rake install
+
+# Interactive console with gem loaded
+bin/console
+```
+
+## Architecture
+
+### Migration Flow
+
+```
+Migration.perform! → MasterProcess.run! → launches tasks in threads
+  SimpleTask:   executes up/down blocks directly
+  DocumentTask: iterates documents → batch → execute → write (sequential)
+  ParallelTask: splits into ID batches → enqueues ParallelJob per batch (via ActiveJob)
+```
+
+### Key Components
+
+- **`Hekenga::Migration`** — main migration class, orchestrates tasks
+- **`Hekenga::MasterProcess`** — launches tasks, manages execution/recovery/progress
+- **`Hekenga::DSL::*`** — fluent DSL for defining migrations (`DSL::Migration`, `DSL::SimpleTask`, `DSL::DocumentTask`)
+- **`Hekenga::DocumentTaskExecutor`** — core document processing: filter → up block → validate → write
+- **`Hekenga::ParallelTask`** / **`Hekenga::ParallelJob`** — parallel execution via ActiveJob
+- **`Hekenga::DocumentTaskRecord`** — Mongoid doc tracking parallel task progress
+- **`Hekenga::Log`** — Mongoid doc tracking migration/task status (`:naught`, `:running`, `:complete`, `:failed`, `:skipped`)
+- **`Hekenga::Failure::*`** — error/validation/write/cancelled failure tracking subclasses
+- **`Hekenga::IdIterator`** / **`Hekenga::MongoidIterator`** — efficient document iteration for parallel vs sequential paths
+
+### Task Types
+
+- **SimpleTask** — one-off up/down blocks, no document iteration
+- **DocumentTask** — per-document processing with scope, filter, setup, up, down, after blocks; supports `parallel!`, `timeless!`, `always_write!`, `use_transaction!`, configurable write strategies (`:update` vs `:delete_then_insert`)
+
+### Configuration
+
+Via `Hekenga.configure` block — sets migration directory and report frequency. Thread-safe registry tracks all migrations.
+
+## Dependencies
+
+- **mongoid** (>= 6), **activejob** (>= 5), **thor** (1.2.1)
+- Test: **rspec** (~> 3.0), **database_cleaner-mongoid** (~> 2.0), **pry**

data/README.md

@@ -1,7 +1,7 @@
 # Hekenga
 
-An attempt at a migration framework for MongoDB that supports parallel document
-processing via ActiveJob, chained jobs and error recovery.
+A migration framework for MongoDB (via Mongoid) that supports parallel document
+processing via ActiveJob, chained jobs, and error recovery.
 
 ## Installation
 
@@ -19,13 +19,135 @@ Or install it yourself as:
 
     $ gem install hekenga
 
+## Configuration
+
+```ruby
+Hekenga.configure do |config|
+  config.dir  = ["db", "hekenga"] # where migration files live (relative to root)
+  config.root = Dir.pwd           # application root
+end
+```
+
+Migrations are stored as Ruby files in the configured directory (default: `db/hekenga/`).
+
 ## Usage
 
-CLI instructions:
+### CLI
+
+```
+$ hekenga help                          # Show all available commands
+$ hekenga generate <description>        # Generate a new migration scaffold
+$ hekenga status                        # Show status of all migrations
+$ hekenga run_all!                      # Run all pending migrations in date order
+$ hekenga run! <path_or_pkey>           # Run a specific migration
+$ hekenga run! <path_or_pkey> --test    # Dry run (no writes persisted)
+$ hekenga run! <path_or_pkey> --clear   # Clear logs before running
+$ hekenga recover! <path_or_pkey>       # Re-process failed/invalid records
+$ hekenga cancel                        # Cancel all active migrations
+$ hekenga skip <path_or_pkey>           # Mark a migration as skipped
+$ hekenga clear! <path_or_pkey>         # Remove all logs/failures for a migration
+$ hekenga cleanup                       # Remove all failure logs
+```
+
+### Writing Migrations
+
+Generate a migration scaffold:
+
+    $ hekenga generate "Add default role to users"
+
+#### Simple Tasks
+
+Simple tasks run arbitrary code once. Use `actual?` and `test?` to check execution mode.
+
+```ruby
+Hekenga.migration do
+  description "Backfill analytics collection"
+  created "2024-01-15 10:00"
+
+  task "Create indexes" do
+    up do
+      Analytics.create_indexes if actual?
+    end
+  end
+end
+```
+
+#### Document Tasks
+
+Document tasks iterate over a Mongoid scope and process each document in batches.
+
+```ruby
+Hekenga.migration do
+  description "Normalize user emails"
+  created "2024-01-15 10:00"
+  batch_size 100 # default batch size for all tasks in this migration
+
+  per_document "Downcase emails" do
+    scope User.all
+
+    # Called once per batch; instance variables are shared with filter/up/after
+    setup do |docs|
+      @domain_map = ExternalService.load_domains
+    end
+
+    # Return false to skip a document
+    filter do |doc|
+      doc.email.present?
+    end
+
+    # Mutate the document in place — Hekenga handles persistence
+    up do |doc|
+      doc.email = doc.email.downcase
+    end
+
+    # Called once per batch with the successfully written documents
+    after do |docs|
+      AuditLog.record(docs.map(&:id))
+    end
+  end
+end
+```
+
+#### Document Task Options
+
+```ruby
+per_document "Process records" do
+  scope MyModel.where(active: true)
+
+  parallel!                           # Process batches in parallel via ActiveJob
+  timeless!                           # Don't update Mongoid timestamps
+  always_write!                       # Write even if the document didn't change
+  skip_prepare!                       # Skip Mongoid callbacks on load
+  use_transaction!                    # Wrap each batch in a MongoDB transaction
+  batch_size 50                       # Override migration-level batch size
+  write_strategy :update              # :update (default) or :delete_then_insert
+  cursor_timeout 86_400               # Max cursor lifetime in seconds (default: 1 day)
+
+  up do |doc|
+    doc.status = "migrated"
+  end
+end
+```
+
+### Test Mode
+
+Run a migration without persisting changes:
+
+```ruby
+migration = Hekenga.find_migration("2024-01-15-add-default-role-to-users")
+migration.test_mode!
+migration.perform!
+```
+
+Or via the CLI:
+
+    $ hekenga run! <path_or_pkey> --test
+
+### Recovery
 
-    $ hekenga help
+When a migration fails (due to errors, invalid records, or write failures), Hekenga logs the failures and marks the migration as failed. You can re-process only the failed records:
 
-Migration DSL documentation TBD, for now please look at spec/
+    $ hekenga recover! <path_or_pkey>
 
 ## Development
 

data/lib/hekenga/document_task.rb

@@ -24,6 +24,9 @@ module Hekenga
 
     def validate!
       raise Hekenga::Invalid.new(self, :ups, "missing") unless ups.any?
+      if scope&.options&.key?(:fields)
+        raise Hekenga::Invalid.new(self, :scope, "uses .only() or .without() which would cause data loss with replace_one")
+      end
     end
 
     def up!(context, document)

data/lib/hekenga/document_task_executor.rb

@@ -59,7 +59,11 @@ module Hekenga
     end
 
     def record_scope
-      task.scope.klass.unscoped.in(_id: task_record.ids)
+      scope = task.scope.klass.unscoped.in(_id: task_record.ids)
+      if task.scope.inclusions.any?
+        scope = scope.includes(*task.scope.inclusions.map(&:name))
+      end
+      scope
     end
 
     def records

data/lib/hekenga/version.rb

@@ -1,3 +1,3 @@
 module Hekenga
-  VERSION = "2.0.0"
+  VERSION = "2.1.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hekenga
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.1.0
 platform: ruby
 authors:
 - Tapio Saarinen
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-07-31 00:00:00.000000000 Z
+date: 2026-04-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -148,6 +148,7 @@ files:
 - ".rspec"
 - ".travis.yml"
 - CHANGELOG.md
+- CLAUDE.md
 - Gemfile
 - README.md
 - Rakefile