data_shifter 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +14 -0
- data/README.md +44 -2
- data/lib/data_shifter/internal/colors.rb +71 -0
- data/lib/data_shifter/internal/output.rb +106 -71
- data/lib/data_shifter/shift.rb +110 -4
- data/lib/data_shifter/version.rb +1 -1
- data/lib/generators/data_shift_generator.rb +91 -17
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7c2c6cb1c13bba3100efe294ceeafd838f4c329650b323457b0a75296bbff28f
|
|
4
|
+
data.tar.gz: b2dfe9b104bcc97fe7f8d1524d9739cb6b5918885a64f0ccf58725a8477d4298
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c501bef9515dae1a53a20dd4b40e4fb454c31b17a781b53df02b861f6ae0415f012a801f2b767f5975cd8cb5aa69555467de5a89dff4448b766ab410e23e7a5a
|
|
7
|
+
data.tar.gz: cd30ac7dcef93f800101c26a5472063859e8671270c58169083c7116fdd3cac2f6559efe6f9da084a2b7b939aa2ef0e909a96b816964043f8739d544a80fef34
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,20 @@
|
|
|
4
4
|
|
|
5
5
|
* N/A
|
|
6
6
|
|
|
7
|
+
## [0.3.0]
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- **Task-based shifts**: New `task` DSL for targeted, one-off changes without the `collection`/`process_record` pattern. Define one or more `task "label" do ... end` blocks that run in sequence with shared transaction and dry-run semantics. Labels appear in output and error messages.
|
|
12
|
+
- **Generator `--task` option**: `rails g data_shift fix_order_1234 --task` generates a shift with a `task` block instead of `collection`/`process_record`.
|
|
13
|
+
- **Colorized CLI output**: Headers, summaries, and status output now use ANSI colors for better readability. Colors are automatically disabled when output is not a TTY or when `NO_COLOR` environment variable is set.
|
|
14
|
+
- **Cleaner summaries**: `Failed` and `Skipped` lines are now omitted from summaries when their values are zero.
|
|
15
|
+
|
|
16
|
+
### Changed
|
|
17
|
+
|
|
18
|
+
- **Improved error messages**: `NotImplementedError` messages for `collection` and `process_record` now suggest using `task` blocks as an alternative.
|
|
19
|
+
- **Task labels logged on execution**: When running task-based shifts, each labeled task logs its name (`>> label`) when it starts.
|
|
20
|
+
|
|
7
21
|
## [0.2.0]
|
|
8
22
|
|
|
9
23
|
### Added
|
data/README.md
CHANGED
|
@@ -35,7 +35,9 @@ COMMIT=1 rake data:shift:backfill_foo
|
|
|
35
35
|
|
|
36
36
|
## Defining a shift
|
|
37
37
|
|
|
38
|
-
|
|
38
|
+
### Collection-based shifts (typical)
|
|
39
|
+
|
|
40
|
+
For systemic migrations across many records, implement:
|
|
39
41
|
|
|
40
42
|
- **`collection`**: an `ActiveRecord::Relation` (uses `find_each`) or an `Array`/Enumerable
|
|
41
43
|
- **`process_record(record)`**: applies the change for one record
|
|
@@ -56,9 +58,48 @@ module DataShifts
|
|
|
56
58
|
end
|
|
57
59
|
```
|
|
58
60
|
|
|
61
|
+
### Task-based shifts (targeted, one-off changes)
|
|
62
|
+
|
|
63
|
+
For targeted changes to specific records (e.g. fixing a bug for particular IDs), use `task` blocks instead:
|
|
64
|
+
|
|
65
|
+
```ruby
|
|
66
|
+
module DataShifts
|
|
67
|
+
class FixOrderDiscrepancies < DataShifter::Shift
|
|
68
|
+
description "Fix order #1234 shipping and billing issues"
|
|
69
|
+
|
|
70
|
+
task "Correct shipping address" do
|
|
71
|
+
order.update!(shipping_address: "123 Main St")
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
task "Apply missing discount" do
|
|
75
|
+
order.update!(discount_cents: 500)
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
private
|
|
79
|
+
|
|
80
|
+
def order
|
|
81
|
+
@order ||= Order.find(1234)
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Task blocks run in the context of the shift instance, so they have access to private helper methods, `dry_run?`, `log`, `skip!`, `find_exactly!`, and any other instance methods you define. Use private methods to DRY up shared lookups across tasks.
|
|
88
|
+
|
|
89
|
+
Task blocks:
|
|
90
|
+
|
|
91
|
+
- Run in sequence within the same lifecycle (transaction, dry run protection, summary)
|
|
92
|
+
- Default to single transaction (all tasks commit or roll back together); use `transaction :per_record` for per-task transactions
|
|
93
|
+
|
|
94
|
+
Generate a task-based shift with:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
bin/rails generate data_shift fix_order_1234 --task
|
|
98
|
+
```
|
|
99
|
+
|
|
59
100
|
## Dry run vs commit
|
|
60
101
|
|
|
61
|
-
Shifts run in **dry run** mode by default.
|
|
102
|
+
Shifts run in **dry run** mode by default. DB changes are always rolled back in dry run mode, regardless of transaction setting.
|
|
62
103
|
|
|
63
104
|
- **Dry run (default)**: `rake data:shift:backfill_foo`
|
|
64
105
|
- **Commit**: `COMMIT=1 rake data:shift:backfill_foo`
|
|
@@ -277,6 +318,7 @@ end
|
|
|
277
318
|
| `bin/rails generate data_shift backfill_foo` | `lib/data_shifts/<timestamp>_backfill_foo.rb` with a `DataShifts::BackfillFoo` class |
|
|
278
319
|
| `bin/rails generate data_shift backfill_users --model User` | Same, with `User.all` in `collection` and `process_record(user)` |
|
|
279
320
|
| `bin/rails generate data_shift backfill_users --spec` | Also generates `spec/lib/data_shifts/backfill_users_spec.rb` when RSpec is enabled |
|
|
321
|
+
| `bin/rails generate data_shift fix_order_1234 --task` | Generates a shift with a `task` block instead of `collection`/`process_record` |
|
|
280
322
|
|
|
281
323
|
The generator refuses to create a second shift if it would produce a duplicate rake task name.
|
|
282
324
|
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module DataShifter
|
|
4
|
+
module Internal
|
|
5
|
+
# ANSI color utilities for CLI output.
|
|
6
|
+
# Automatically detects TTY and respects NO_COLOR environment variable.
|
|
7
|
+
module Colors
|
|
8
|
+
CODES = {
|
|
9
|
+
reset: "\e[0m",
|
|
10
|
+
bold: "\e[1m",
|
|
11
|
+
dim: "\e[2m",
|
|
12
|
+
green: "\e[32m",
|
|
13
|
+
yellow: "\e[33m",
|
|
14
|
+
red: "\e[31m",
|
|
15
|
+
cyan: "\e[36m",
|
|
16
|
+
}.freeze
|
|
17
|
+
|
|
18
|
+
module_function
|
|
19
|
+
|
|
20
|
+
def enabled?(io = $stdout)
|
|
21
|
+
return false if ENV["NO_COLOR"]
|
|
22
|
+
return false unless io.respond_to?(:tty?)
|
|
23
|
+
|
|
24
|
+
io.tty?
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def wrap(text, *styles, io: $stdout)
|
|
28
|
+
return text unless enabled?(io)
|
|
29
|
+
|
|
30
|
+
codes = styles.map { |s| CODES[s] }.compact.join
|
|
31
|
+
"#{codes}#{text}#{CODES[:reset]}"
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def bold(text, io: $stdout)
|
|
35
|
+
wrap(text, :bold, io:)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def dim(text, io: $stdout)
|
|
39
|
+
wrap(text, :dim, io:)
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def green(text, io: $stdout)
|
|
43
|
+
wrap(text, :green, io:)
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def yellow(text, io: $stdout)
|
|
47
|
+
wrap(text, :yellow, io:)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def red(text, io: $stdout)
|
|
51
|
+
wrap(text, :red, io:)
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def cyan(text, io: $stdout)
|
|
55
|
+
wrap(text, :cyan, io:)
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def success(text, io: $stdout)
|
|
59
|
+
wrap(text, :bold, :green, io:)
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def warning(text, io: $stdout)
|
|
63
|
+
wrap(text, :bold, :yellow, io:)
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def error(text, io: $stdout)
|
|
67
|
+
wrap(text, :bold, :red, io:)
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
end
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require_relative "colors"
|
|
4
|
+
|
|
3
5
|
module DataShifter
|
|
4
6
|
module Internal
|
|
5
7
|
# Output formatting utilities for data shift runs.
|
|
@@ -12,131 +14,164 @@ module DataShifter
|
|
|
12
14
|
}.freeze
|
|
13
15
|
|
|
14
16
|
SKIP_REASONS_DISPLAY_LIMIT = 10
|
|
17
|
+
DIVIDER = "=" * 60
|
|
18
|
+
SEPARATOR = "-" * 60
|
|
15
19
|
|
|
16
20
|
module_function
|
|
17
21
|
|
|
22
|
+
# --- Public header methods ---
|
|
23
|
+
|
|
18
24
|
def print_header(io:, shift_class:, total:, label:, dry_run:, transaction_mode:, status_interval:)
|
|
19
|
-
io
|
|
20
|
-
io.puts "=" * 60
|
|
21
|
-
io.puts shift_class.name || "DataShifter::Shift (anonymous)"
|
|
22
|
-
io.puts "\"#{shift_class.description}\"" if shift_class.description.present?
|
|
23
|
-
io.puts "-" * 60
|
|
24
|
-
io.puts "Mode: #{dry_run ? "DRY RUN (no changes will be persisted)" : "LIVE"}"
|
|
25
|
+
print_header_top(io:, shift_class:, dry_run:)
|
|
25
26
|
io.puts "Records: #{total} #{label}"
|
|
26
27
|
io.puts "Transaction: #{TRANSACTION_MODE_LABELS[transaction_mode]}"
|
|
28
|
+
print_header_bottom(io:, status_interval:)
|
|
29
|
+
end
|
|
27
30
|
|
|
28
|
-
|
|
29
|
-
io
|
|
30
|
-
|
|
31
|
-
io.puts "
|
|
32
|
-
io
|
|
31
|
+
def print_task_header(io:, shift_class:, block_count:, dry_run:, transaction_mode:, status_interval:)
|
|
32
|
+
print_header_top(io:, shift_class:, dry_run:)
|
|
33
|
+
io.puts "Tasks: #{block_count}" if block_count >= 2
|
|
34
|
+
io.puts "Transaction: #{task_transaction_label(transaction_mode)}"
|
|
35
|
+
print_header_bottom(io:, status_interval:)
|
|
33
36
|
end
|
|
34
37
|
|
|
38
|
+
# --- Public summary/progress methods ---
|
|
39
|
+
|
|
35
40
|
def print_summary(io:, stats:, errors:, start_time:, dry_run:, transaction_mode:, interrupted:, task_name:, last_successful_id:, skip_reasons: {})
|
|
36
41
|
return unless start_time
|
|
37
42
|
|
|
38
|
-
|
|
43
|
+
has_failures = stats[:failed].positive? || interrupted
|
|
44
|
+
|
|
39
45
|
io.puts ""
|
|
40
|
-
io.puts
|
|
41
|
-
io.puts summary_title(dry_run:, interrupted:)
|
|
42
|
-
io.puts
|
|
43
|
-
io
|
|
44
|
-
io.puts "Processed: #{stats[:processed]}"
|
|
45
|
-
io.puts "Succeeded: #{stats[:succeeded]}"
|
|
46
|
-
io.puts "Failed: #{stats[:failed]}"
|
|
47
|
-
io.puts "Skipped: #{stats[:skipped]}"
|
|
48
|
-
print_skip_reasons(io:, skip_reasons:) if skip_reasons.any?
|
|
46
|
+
io.puts summary_divider(has_failures:, io:)
|
|
47
|
+
io.puts summary_title(dry_run:, interrupted:, has_failures:, io:)
|
|
48
|
+
io.puts Colors.dim(SEPARATOR, io:)
|
|
49
|
+
print_stats(io:, stats:, start_time:, skip_reasons:)
|
|
49
50
|
|
|
50
51
|
print_errors(io:, errors:) if errors.any?
|
|
51
52
|
print_interrupt_warning(io:, transaction_mode:, dry_run:) if interrupted
|
|
52
53
|
print_dry_run_instructions(io:, task_name:) if dry_run && !interrupted
|
|
53
54
|
print_continue_from_hint(io:, task_name:, last_successful_id:, dry_run:, transaction_mode:, errors:)
|
|
54
55
|
|
|
55
|
-
io.puts
|
|
56
|
+
io.puts summary_divider(has_failures:, io:)
|
|
56
57
|
end
|
|
57
58
|
|
|
58
59
|
def print_progress(io:, stats:, errors:, start_time:, status_interval:, skip_reasons: {})
|
|
59
60
|
return unless start_time
|
|
60
61
|
|
|
61
|
-
elapsed = (Time.current - start_time).round(1)
|
|
62
62
|
io.puts ""
|
|
63
|
-
io.puts
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
elsif Signal.list.key?("INFO")
|
|
68
|
-
"Ctrl+T"
|
|
69
|
-
else
|
|
70
|
-
"SIGUSR1"
|
|
71
|
-
end
|
|
72
|
-
|
|
73
|
-
io.puts "STATUS (still running) — triggered by #{trigger}"
|
|
74
|
-
io.puts "-" * 60
|
|
75
|
-
io.puts "Duration: #{elapsed}s"
|
|
76
|
-
io.puts "Processed: #{stats[:processed]}"
|
|
77
|
-
io.puts "Succeeded: #{stats[:succeeded]}"
|
|
78
|
-
io.puts "Failed: #{stats[:failed]}"
|
|
79
|
-
io.puts "Skipped: #{stats[:skipped]}"
|
|
80
|
-
print_skip_reasons(io:, skip_reasons:) if skip_reasons.any?
|
|
63
|
+
io.puts Colors.cyan(DIVIDER, io:)
|
|
64
|
+
io.puts "#{Colors.cyan("STATUS (still running)", io:)} — triggered by #{status_trigger(status_interval)}"
|
|
65
|
+
io.puts Colors.dim(SEPARATOR, io:)
|
|
66
|
+
print_stats(io:, stats:, start_time:, skip_reasons:)
|
|
81
67
|
|
|
82
68
|
print_errors(io:, errors:) if errors.any?
|
|
83
69
|
|
|
84
|
-
io.puts
|
|
70
|
+
io.puts Colors.cyan(DIVIDER, io:)
|
|
85
71
|
io.puts ""
|
|
86
72
|
end
|
|
87
73
|
|
|
88
74
|
def print_errors(io:, errors:)
|
|
89
75
|
io.puts ""
|
|
90
|
-
io.puts "ERRORS:"
|
|
91
|
-
errors.each
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
76
|
+
io.puts Colors.error("ERRORS:", io:)
|
|
77
|
+
errors.each { |err| print_single_error(io:, err:) }
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
# --- Private helpers ---
|
|
81
|
+
|
|
82
|
+
def print_header_top(io:, shift_class:, dry_run:)
|
|
83
|
+
io.puts ""
|
|
84
|
+
io.puts Colors.dim(DIVIDER, io:)
|
|
85
|
+
io.puts Colors.bold(shift_class.name || "DataShifter::Shift (anonymous)", io:)
|
|
86
|
+
io.puts Colors.dim("\"#{shift_class.description}\"", io:) if shift_class.description.present?
|
|
87
|
+
io.puts Colors.dim(SEPARATOR, io:)
|
|
88
|
+
io.puts "Mode: #{mode_label(dry_run:, io:)}"
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def print_header_bottom(io:, status_interval:)
|
|
92
|
+
status_line = build_status_line(status_interval)
|
|
93
|
+
io.puts Colors.dim("Status: #{status_line} for live progress (no abort)", io:) if status_line
|
|
94
|
+
io.puts Colors.dim(DIVIDER, io:)
|
|
95
|
+
io.puts ""
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def print_stats(io:, stats:, start_time:, skip_reasons:)
|
|
99
|
+
elapsed = (Time.current - start_time).round(1)
|
|
100
|
+
io.puts "Duration: #{elapsed}s"
|
|
101
|
+
io.puts "Processed: #{stats[:processed]}"
|
|
102
|
+
io.puts "Succeeded: #{Colors.green(stats[:succeeded].to_s, io:)}"
|
|
103
|
+
io.puts "Failed: #{Colors.red(stats[:failed].to_s, io:)}" if stats[:failed].positive?
|
|
104
|
+
io.puts "Skipped: #{Colors.yellow(stats[:skipped].to_s, io:)}" if stats[:skipped].positive?
|
|
105
|
+
print_skip_reasons(io:, skip_reasons:) if skip_reasons.any?
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def print_single_error(io:, err:)
|
|
109
|
+
lines = err[:error].to_s.split("\n")
|
|
110
|
+
io.puts " #{Colors.red(err[:record].to_s, io:)}: #{lines.first}"
|
|
111
|
+
lines.drop(1).each { |line| io.puts " #{line}" }
|
|
112
|
+
err[:backtrace]&.each { |line| io.puts Colors.dim(" #{line}", io:) }
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def mode_label(dry_run:, io:)
|
|
116
|
+
if dry_run
|
|
117
|
+
"#{Colors.cyan("DRY RUN", io:)} (no changes will be persisted)"
|
|
118
|
+
else
|
|
119
|
+
Colors.warning("LIVE", io:)
|
|
96
120
|
end
|
|
97
121
|
end
|
|
98
122
|
|
|
99
|
-
def
|
|
123
|
+
def task_transaction_label(mode)
|
|
124
|
+
mode == :per_record ? "per-task" : TRANSACTION_MODE_LABELS[mode]
|
|
125
|
+
end
|
|
126
|
+
|
|
127
|
+
def summary_divider(has_failures:, io:)
|
|
128
|
+
has_failures ? Colors.red(DIVIDER, io:) : Colors.green(DIVIDER, io:)
|
|
129
|
+
end
|
|
130
|
+
|
|
131
|
+
def summary_title(dry_run:, interrupted:, has_failures: false, io: $stdout)
|
|
100
132
|
base = dry_run ? "SUMMARY (DRY RUN)" : "SUMMARY"
|
|
101
|
-
interrupted ? "#{base} - INTERRUPTED" : base
|
|
133
|
+
title = interrupted ? "#{base} - INTERRUPTED" : base
|
|
134
|
+
has_failures ? Colors.error(title, io:) : Colors.success(title, io:)
|
|
102
135
|
end
|
|
103
136
|
|
|
104
|
-
def
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
io.puts " Non-DB side effects (API calls, emails, etc.) are not rolled back."
|
|
110
|
-
io.puts " Review the database state before re-running."
|
|
111
|
-
elsif dry_run
|
|
112
|
-
io.puts "[!] INTERRUPTED: All DB changes have been rolled back (dry run)."
|
|
113
|
-
io.puts " Non-DB side effects (API calls, emails, etc.) are not rolled back."
|
|
137
|
+
def status_trigger(status_interval)
|
|
138
|
+
if status_interval
|
|
139
|
+
"every #{status_interval}s (STATUS_INTERVAL)"
|
|
140
|
+
elsif Signal.list.key?("INFO")
|
|
141
|
+
"Ctrl+T"
|
|
114
142
|
else
|
|
115
|
-
|
|
116
|
-
io.puts " No DB changes were persisted."
|
|
117
|
-
io.puts " Non-DB side effects (API calls, emails, etc.) are not rolled back."
|
|
143
|
+
"SIGUSR1"
|
|
118
144
|
end
|
|
119
145
|
end
|
|
120
146
|
|
|
147
|
+
def print_interrupt_warning(io:, transaction_mode:, dry_run:)
|
|
148
|
+
msg = if transaction_mode == :none
|
|
149
|
+
"`transaction false` mode was active. Some DB changes may have been applied."
|
|
150
|
+
elsif dry_run
|
|
151
|
+
"All DB changes have been rolled back (dry run)."
|
|
152
|
+
else
|
|
153
|
+
"DB transaction has been rolled back. No DB changes were persisted."
|
|
154
|
+
end
|
|
155
|
+
io.puts ""
|
|
156
|
+
io.puts "#{Colors.warning("[!] INTERRUPTED:", io:)} #{msg}"
|
|
157
|
+
io.puts " Non-DB side effects (API calls, emails, etc.) are not rolled back."
|
|
158
|
+
end
|
|
159
|
+
|
|
121
160
|
def print_dry_run_instructions(io:, task_name:)
|
|
122
161
|
io.puts ""
|
|
123
|
-
io.puts "[!] No changes were saved."
|
|
162
|
+
io.puts Colors.cyan("[!] No changes were saved.", io:)
|
|
124
163
|
return unless task_name.present?
|
|
125
164
|
|
|
126
165
|
io.puts "To apply these changes, run:"
|
|
127
|
-
io.puts " COMMIT=1 rake data:shift:#{task_name}"
|
|
166
|
+
io.puts " #{Colors.bold("COMMIT=1 rake data:shift:#{task_name}", io:)}"
|
|
128
167
|
end
|
|
129
168
|
|
|
130
169
|
def print_continue_from_hint(io:, task_name:, last_successful_id:, dry_run:, transaction_mode:, errors:)
|
|
131
|
-
return if dry_run
|
|
132
|
-
return unless transaction_mode == :none
|
|
133
|
-
return if errors.empty?
|
|
134
|
-
return unless last_successful_id
|
|
135
|
-
return unless task_name.present?
|
|
170
|
+
return if dry_run || transaction_mode != :none || errors.empty? || !last_successful_id || !task_name.present?
|
|
136
171
|
|
|
137
172
|
io.puts ""
|
|
138
173
|
io.puts "To resume from the last successful record:"
|
|
139
|
-
io.puts " CONTINUE_FROM=#{last_successful_id} COMMIT=1 rake data:shift:#{task_name}"
|
|
174
|
+
io.puts " #{Colors.bold("CONTINUE_FROM=#{last_successful_id} COMMIT=1 rake data:shift:#{task_name}", io:)}"
|
|
140
175
|
end
|
|
141
176
|
|
|
142
177
|
def build_status_line(status_interval)
|
data/lib/data_shifter/shift.rb
CHANGED
|
@@ -9,6 +9,7 @@ require_relative "internal/record_utils"
|
|
|
9
9
|
require_relative "internal/progress_bar"
|
|
10
10
|
require_relative "internal/side_effect_guards"
|
|
11
11
|
require_relative "internal/log_deduplicator"
|
|
12
|
+
require_relative "internal/colors"
|
|
12
13
|
|
|
13
14
|
# Base class for data shifts. Dry-run by default, progress bars, transaction modes, consistent summaries.
|
|
14
15
|
#
|
|
@@ -67,6 +68,7 @@ module DataShifter
|
|
|
67
68
|
class_attribute :_throttle_interval, default: nil
|
|
68
69
|
class_attribute :_allow_external_requests, default: [], instance_accessor: false
|
|
69
70
|
class_attribute :_suppress_repeated_logs, default: nil, instance_accessor: false
|
|
71
|
+
class_attribute :_task_blocks, default: [], instance_accessor: false
|
|
70
72
|
|
|
71
73
|
# Internal exception used by skip! to abort the current process_record.
|
|
72
74
|
# Rescued in _process_one; not propagated.
|
|
@@ -128,6 +130,21 @@ module DataShifter
|
|
|
128
130
|
self._suppress_repeated_logs = !!enabled
|
|
129
131
|
end
|
|
130
132
|
|
|
133
|
+
# Define a task block to run instead of collection/process_record.
|
|
134
|
+
# Multiple blocks run in sequence; labels appear in errors and summary.
|
|
135
|
+
# Example:
|
|
136
|
+
# task "Fix user A" do
|
|
137
|
+
# User.find(123).update!(...)
|
|
138
|
+
# end
|
|
139
|
+
# task "Fix user B" do
|
|
140
|
+
# User.find(456).update!(...)
|
|
141
|
+
# end
|
|
142
|
+
def task(label = nil, &block)
|
|
143
|
+
raise ArgumentError, "task requires a block" unless block_given?
|
|
144
|
+
|
|
145
|
+
self._task_blocks = (_task_blocks || []).dup + [{ label: label.presence, block: }]
|
|
146
|
+
end
|
|
147
|
+
|
|
131
148
|
def run!
|
|
132
149
|
dry_run = Internal::Env.dry_run?
|
|
133
150
|
result = call(dry_run:)
|
|
@@ -139,7 +156,11 @@ module DataShifter
|
|
|
139
156
|
# --- Public API (intentionally exposed to subclasses) ---
|
|
140
157
|
|
|
141
158
|
def call
|
|
142
|
-
|
|
159
|
+
if self.class._task_blocks.any?
|
|
160
|
+
_run_task_blocks
|
|
161
|
+
else
|
|
162
|
+
_for_each_record_in(collection) { |record| process_record(record) }
|
|
163
|
+
end
|
|
143
164
|
end
|
|
144
165
|
|
|
145
166
|
def find_exactly!(model, ids)
|
|
@@ -163,7 +184,7 @@ module DataShifter
|
|
|
163
184
|
end
|
|
164
185
|
|
|
165
186
|
def log(message)
|
|
166
|
-
puts message
|
|
187
|
+
puts Internal::Colors.dim(message)
|
|
167
188
|
end
|
|
168
189
|
|
|
169
190
|
private
|
|
@@ -259,11 +280,13 @@ module DataShifter
|
|
|
259
280
|
# --- Override points ---
|
|
260
281
|
|
|
261
282
|
def collection
|
|
262
|
-
raise NotImplementedError,
|
|
283
|
+
raise NotImplementedError,
|
|
284
|
+
"#{self.class.name}: override `collection` (or use one or more `task 'label' do ... end` blocks for targeted shifts)."
|
|
263
285
|
end
|
|
264
286
|
|
|
265
287
|
def process_record(_record)
|
|
266
|
-
raise NotImplementedError,
|
|
288
|
+
raise NotImplementedError,
|
|
289
|
+
"#{self.class.name}: override `process_record` (or use one or more `task 'label' do ... end` blocks for targeted shifts)."
|
|
267
290
|
end
|
|
268
291
|
|
|
269
292
|
# --- Record iteration ---
|
|
@@ -452,5 +475,88 @@ module DataShifter
|
|
|
452
475
|
# Re-raise to trigger transaction rollback in the wrapping transaction block
|
|
453
476
|
raise Interrupt
|
|
454
477
|
end
|
|
478
|
+
|
|
479
|
+
# --- Task block execution ---
|
|
480
|
+
|
|
481
|
+
def _run_task_blocks
|
|
482
|
+
_validate_no_collection_or_process_record_override!
|
|
483
|
+
|
|
484
|
+
blocks = self.class._task_blocks
|
|
485
|
+
_reset_tracking
|
|
486
|
+
_print_task_header(blocks.size)
|
|
487
|
+
|
|
488
|
+
ActiveSupport::IsolatedExecutionState[:_data_shifter_current_run] = self
|
|
489
|
+
status_proc = proc { ActiveSupport::IsolatedExecutionState[:_data_shifter_current_run]&.send(:_print_progress) }
|
|
490
|
+
prev_handlers = Internal::SignalHandler.install_status_traps(status_proc)
|
|
491
|
+
|
|
492
|
+
begin
|
|
493
|
+
blocks.each do |entry|
|
|
494
|
+
_execute_task_block(entry)
|
|
495
|
+
end
|
|
496
|
+
fail! "#{@stats[:failed]} task(s) failed" if @errors.any?
|
|
497
|
+
rescue Interrupt
|
|
498
|
+
_handle_interrupt
|
|
499
|
+
ensure
|
|
500
|
+
ActiveSupport::IsolatedExecutionState.delete(:_data_shifter_current_run)
|
|
501
|
+
Internal::SignalHandler.restore_status_traps(prev_handlers)
|
|
502
|
+
end
|
|
503
|
+
end
|
|
504
|
+
|
|
505
|
+
def _validate_no_collection_or_process_record_override!
|
|
506
|
+
collection_overridden = instance_method_owner(:collection) != DataShifter::Shift
|
|
507
|
+
process_record_overridden = instance_method_owner(:process_record) != DataShifter::Shift
|
|
508
|
+
|
|
509
|
+
return unless collection_overridden || process_record_overridden
|
|
510
|
+
|
|
511
|
+
raise ArgumentError,
|
|
512
|
+
"Cannot use task blocks and override collection or process_record; use one mode or the other."
|
|
513
|
+
end
|
|
514
|
+
|
|
515
|
+
def instance_method_owner(method_name)
|
|
516
|
+
self.class.instance_method(method_name).owner
|
|
517
|
+
end
|
|
518
|
+
|
|
519
|
+
def _execute_task_block(entry)
|
|
520
|
+
label = entry[:label]
|
|
521
|
+
block = entry[:block]
|
|
522
|
+
|
|
523
|
+
if _transaction_mode == :per_record && !dry_run?
|
|
524
|
+
::ActiveRecord::Base.transaction do
|
|
525
|
+
_run_single_task_block(label, block)
|
|
526
|
+
end
|
|
527
|
+
else
|
|
528
|
+
_run_single_task_block(label, block)
|
|
529
|
+
end
|
|
530
|
+
end
|
|
531
|
+
|
|
532
|
+
def _run_single_task_block(label, block)
|
|
533
|
+
puts Internal::Colors.cyan(">> #{label} <<") if label.present?
|
|
534
|
+
instance_exec(&block)
|
|
535
|
+
@stats[:processed] += 1
|
|
536
|
+
@stats[:succeeded] += 1
|
|
537
|
+
rescue SkipRecord
|
|
538
|
+
# skip! already updated @stats[:skipped] and @skip_reasons; continue to next block
|
|
539
|
+
nil
|
|
540
|
+
rescue StandardError => e
|
|
541
|
+
@stats[:failed] += 1
|
|
542
|
+
identifier = label || "task"
|
|
543
|
+
error_text = _format_error(e)
|
|
544
|
+
@errors << { record: identifier, error: error_text, backtrace: e.backtrace&.first(3) }
|
|
545
|
+
_log_error(identifier, error_text)
|
|
546
|
+
|
|
547
|
+
new_message = label.present? ? "#{label}: #{e.message}" : e.message
|
|
548
|
+
raise e.class, new_message, e.backtrace
|
|
549
|
+
end
|
|
550
|
+
|
|
551
|
+
def _print_task_header(block_count)
|
|
552
|
+
Internal::Output.print_task_header(
|
|
553
|
+
io: $stdout,
|
|
554
|
+
shift_class: self.class,
|
|
555
|
+
block_count:,
|
|
556
|
+
dry_run: dry_run?,
|
|
557
|
+
transaction_mode: _transaction_mode,
|
|
558
|
+
status_interval: Internal::Env.status_interval_seconds,
|
|
559
|
+
)
|
|
560
|
+
end
|
|
455
561
|
end
|
|
456
562
|
end
|
data/lib/data_shifter/version.rb
CHANGED
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
# Usage:
|
|
6
6
|
# rails g data_shift backfill_users
|
|
7
7
|
# rails g data_shift backfill_users --model=User
|
|
8
|
+
# rails g data_shift fix_order_1234 --task
|
|
8
9
|
#
|
|
9
10
|
class DataShiftGenerator < Rails::Generators::NamedBase
|
|
10
11
|
class_option :model,
|
|
@@ -17,6 +18,11 @@ class DataShiftGenerator < Rails::Generators::NamedBase
|
|
|
17
18
|
default: false,
|
|
18
19
|
desc: "Generate RSpec file"
|
|
19
20
|
|
|
21
|
+
class_option :task,
|
|
22
|
+
type: :boolean,
|
|
23
|
+
default: false,
|
|
24
|
+
desc: "Generate a task-based shift (uses task blocks instead of collection/process_record)"
|
|
25
|
+
|
|
20
26
|
def check_for_naming_conflict
|
|
21
27
|
underscored_name = name.underscore
|
|
22
28
|
|
|
@@ -44,6 +50,29 @@ class DataShiftGenerator < Rails::Generators::NamedBase
|
|
|
44
50
|
model_name_raw = options[:model].to_s.strip
|
|
45
51
|
@model_name = model_name_raw.present? ? model_name_raw.underscore.singularize.camelize : nil
|
|
46
52
|
|
|
53
|
+
if options[:task]
|
|
54
|
+
_create_task_shift_file(underscored_name)
|
|
55
|
+
else
|
|
56
|
+
_create_standard_shift_file(underscored_name)
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
def create_spec_file
|
|
61
|
+
return unless options[:spec]
|
|
62
|
+
return unless rspec_enabled?
|
|
63
|
+
|
|
64
|
+
underscored_name = name.underscore
|
|
65
|
+
|
|
66
|
+
if options[:task]
|
|
67
|
+
_create_task_spec_file(underscored_name)
|
|
68
|
+
else
|
|
69
|
+
_create_standard_spec_file(underscored_name)
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
private
|
|
74
|
+
|
|
75
|
+
def _create_standard_shift_file(underscored_name)
|
|
47
76
|
collection_body = if @model_name.present?
|
|
48
77
|
"#{@model_name}.all"
|
|
49
78
|
else
|
|
@@ -76,14 +105,34 @@ class DataShiftGenerator < Rails::Generators::NamedBase
|
|
|
76
105
|
RUBY
|
|
77
106
|
end
|
|
78
107
|
|
|
79
|
-
def
|
|
80
|
-
|
|
81
|
-
|
|
108
|
+
def _create_task_shift_file(underscored_name)
|
|
109
|
+
model_comment = @model_name.present? ? "# #{@model_name}.find(...).update!(...)" : "# Model.find(...).update!(...)"
|
|
110
|
+
task_label = @class_name.underscore.humanize
|
|
82
111
|
|
|
83
|
-
underscored_name
|
|
84
|
-
|
|
112
|
+
create_file "lib/data_shifts/#{@timestamp}_#{underscored_name}.rb", <<~RUBY
|
|
113
|
+
# frozen_string_literal: true
|
|
114
|
+
|
|
115
|
+
# rake data:shift:#{underscored_name} # Dry run (default)
|
|
116
|
+
# COMMIT=1 rake data:shift:#{underscored_name} # Apply changes
|
|
117
|
+
|
|
118
|
+
module DataShifts
|
|
119
|
+
class #{@class_name} < DataShifter::Shift
|
|
120
|
+
description "TODO: Describe this shift"
|
|
121
|
+
|
|
122
|
+
transaction true # or :per_record for per-task transactions
|
|
85
123
|
|
|
124
|
+
task "#{task_label}" do
|
|
125
|
+
#{model_comment}
|
|
126
|
+
end
|
|
127
|
+
end
|
|
128
|
+
end
|
|
129
|
+
RUBY
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
def _create_standard_spec_file(underscored_name)
|
|
133
|
+
record_arg = @model_name.present? ? @model_name.underscore : "record"
|
|
86
134
|
model_for_change = @model_name.present? ? @model_name : "Model"
|
|
135
|
+
|
|
87
136
|
create_file "spec/lib/data_shifts/#{underscored_name}_spec.rb", <<~RUBY
|
|
88
137
|
# frozen_string_literal: true
|
|
89
138
|
|
|
@@ -98,28 +147,53 @@ class DataShiftGenerator < Rails::Generators::NamedBase
|
|
|
98
147
|
# Set up test records as needed
|
|
99
148
|
# let(:#{record_arg}) { create(:#{record_arg}) }
|
|
100
149
|
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
150
|
+
context "when dry run" do
|
|
151
|
+
subject(:result) { run_data_shift(described_class, dry_run: true) }
|
|
152
|
+
|
|
153
|
+
it "succeeds without persisting changes" do
|
|
154
|
+
expect { result }.not_to change(#{model_for_change}, :count)
|
|
155
|
+
expect(result).to be_ok
|
|
107
156
|
end
|
|
108
157
|
end
|
|
109
158
|
|
|
110
|
-
|
|
159
|
+
context "when commit" do
|
|
160
|
+
subject(:result) { run_data_shift(described_class, commit: true) }
|
|
161
|
+
|
|
111
162
|
it "applies changes" do
|
|
112
|
-
expect
|
|
113
|
-
|
|
114
|
-
expect(result).to be_ok
|
|
115
|
-
end.to change(#{model_for_change}, :count)
|
|
163
|
+
expect { result }.to change(#{model_for_change}, :count)
|
|
164
|
+
expect(result).to be_ok
|
|
116
165
|
end
|
|
117
166
|
end
|
|
118
167
|
end
|
|
119
168
|
RUBY
|
|
120
169
|
end
|
|
121
170
|
|
|
122
|
-
|
|
171
|
+
def _create_task_spec_file(underscored_name)
|
|
172
|
+
create_file "spec/lib/data_shifts/#{underscored_name}_spec.rb", <<~RUBY
|
|
173
|
+
# frozen_string_literal: true
|
|
174
|
+
|
|
175
|
+
require "rails_helper"
|
|
176
|
+
require "data_shifter/spec_helper"
|
|
177
|
+
|
|
178
|
+
RSpec.describe DataShifts::#{@class_name} do
|
|
179
|
+
include DataShifter::SpecHelper
|
|
180
|
+
|
|
181
|
+
before { allow($stdout).to receive(:puts) }
|
|
182
|
+
|
|
183
|
+
context "when dry run" do
|
|
184
|
+
subject(:result) { run_data_shift(described_class, dry_run: true) }
|
|
185
|
+
|
|
186
|
+
it { is_expected.to be_ok }
|
|
187
|
+
end
|
|
188
|
+
|
|
189
|
+
context "when commit" do
|
|
190
|
+
subject(:result) { run_data_shift(described_class, commit: true) }
|
|
191
|
+
|
|
192
|
+
it { is_expected.to be_ok }
|
|
193
|
+
end
|
|
194
|
+
end
|
|
195
|
+
RUBY
|
|
196
|
+
end
|
|
123
197
|
|
|
124
198
|
def rspec_enabled?
|
|
125
199
|
# Check if rspec-rails is available and configured as the test framework
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: data_shifter
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kali Donovan
|
|
@@ -116,6 +116,7 @@ files:
|
|
|
116
116
|
- lib/data_shifter.rb
|
|
117
117
|
- lib/data_shifter/configuration.rb
|
|
118
118
|
- lib/data_shifter/errors.rb
|
|
119
|
+
- lib/data_shifter/internal/colors.rb
|
|
119
120
|
- lib/data_shifter/internal/env.rb
|
|
120
121
|
- lib/data_shifter/internal/log_deduplicator.rb
|
|
121
122
|
- lib/data_shifter/internal/output.rb
|