data_shifter 0.3.3 → 0.3.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -1
- data/README.md +45 -3
- data/lib/data_shifter/internal/env.rb +19 -6
- data/lib/data_shifter/settings.rb +55 -0
- data/lib/data_shifter/shift.rb +115 -19
- data/lib/data_shifter/version.rb +1 -1
- data/lib/data_shifter.rb +1 -20
- metadata +10 -13
- data/.husky/pre-commit +0 -1
- data/.lintstagedrc +0 -3
- data/Rakefile +0 -16
- data/lib/data_shifter/configuration.rb +0 -48
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2b117c23db49ab1654f56e2901792147277a957da4b65e308a893197846d24e4
|
|
4
|
+
data.tar.gz: 26cae4d79c699e8f84e52570bde8de7150e31bc812562197c1ffd9f1e4fdaf58
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6cc508407a213fae51bb740e352b27d8c470496cc7d6608bdf5a209360f55a801e7a60e9028aa62336b2d32475da7c189d9c79215970e950c30dc5577e056b4d
|
|
7
|
+
data.tar.gz: d3d625d0876a3574994ba877cc89e4c698f4ce083ebab597f1b92051122a78047cec201c2794aac7096e1b958d141c15dddc24d24582c6092a1adbb198502370
|
data/CHANGELOG.md
CHANGED
|
@@ -2,7 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
## [Unreleased]
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## [0.3.4]
|
|
6
|
+
|
|
7
|
+
* [Changed] `COMMIT` / `DRY_RUN` are now parsed as real booleans (`1/true/t/yes/y/on` vs `0/false/f/no/n/off`, case- and whitespace-insensitive). **`DRY_RUN=1` now means a dry run** — previously the `DRY_RUN == "true"` compare read `DRY_RUN=1` as "not dry" and committed the shift, which is data loss for a value that is truthy in every other tool. Unrecognized values (e.g. `COMMIT=garbage`) now raise `ArgumentError` instead of silently dry-running. `DRY_RUN=false` / `COMMIT=1` still commit.
|
|
8
|
+
* [Changed] Bumped the minimum Rails (ActiveRecord, ActiveSupport, Railties) to `>= 7.2`, matching upstream `axn`. Its `on_success` now fires via `ActiveRecord.after_all_transactions_commit`, which requires ActiveRecord 7.2+.
|
|
9
|
+
* [Feature] `throttle` now accepts an optional `per:` keyword. When set, the sleep is inserted only after processing that many records instead of after every record (e.g. `throttle 1.second, per: 100` sleeps once per 100 records).
|
|
10
|
+
* [Feature] `task` now accepts an Axn class as sugar for a one-axn task: `task "label", SomeAxn, foo: 1` forwards to `SomeAxn.call!(foo: 1)`, so a helper axn's failure is never silently swallowed. Keyword args are static (evaluated at class-load); use the block form when you need runtime values. Passing both a class and a block raises.
|
|
11
|
+
* [Feature] `inline_csv` reads CSV colocated with the shift after a `__END__` marker, so small data sets can live alongside the code. Returns the data rows (`CSV::Row` objects by default, so `row["id"]` works); options forward to `CSV.parse`. Typically used as `def collection = inline_csv`. `csv` is required lazily (not a hard dependency) — add `gem "csv"` to your Gemfile if you're on Ruby 3.4+ and hit a load error.
|
|
12
|
+
* [Changed] Configuration now rides on the upstream `Axn::Configurable` DSL instead of a hand-rolled config object. The public API is unchanged — `DataShifter.configure { |c| ... }`, `DataShifter.config.x`, and the per-shift `progress`/`suppress_repeated_logs` overrides all keep working. This raises the minimum `axn` to `>= 0.1.0.alpha.5` (the first release shipping `Axn::Configurable`), now depended on from rubygems rather than a git pin.
|
|
13
|
+
* [Changed] Replaced the removed `log_calls false` DSL with `auto_log false` (axn renamed the method on main). The previous `respond_to?(:log_calls)` guard silently no-op'd, re-enabling axn's per-call logging that shifts intend to suppress.
|
|
6
14
|
|
|
7
15
|
## [0.3.3]
|
|
8
16
|
|
data/README.md
CHANGED
|
@@ -58,6 +58,33 @@ module DataShifts
|
|
|
58
58
|
end
|
|
59
59
|
```
|
|
60
60
|
|
|
61
|
+
### Inline CSV data (small data sets)
|
|
62
|
+
|
|
63
|
+
When the data driving a shift is small, you can colocate it with the code after a `__END__` marker instead of keeping a separate file. `inline_csv` parses that section and returns the rows (`CSV::Row` objects by default, so `row["col"]` works):
|
|
64
|
+
|
|
65
|
+
```ruby
|
|
66
|
+
module DataShifts
|
|
67
|
+
class BackfillTimeZones < DataShifter::Shift
|
|
68
|
+
description "Set time zones from a fixed list"
|
|
69
|
+
|
|
70
|
+
def collection = inline_csv
|
|
71
|
+
|
|
72
|
+
def process_record(row)
|
|
73
|
+
User.find(row["id"]).update!(time_zone: row["time_zone"])
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
__END__
|
|
79
|
+
id,time_zone
|
|
80
|
+
1,Pacific Time (US & Canada)
|
|
81
|
+
2,Eastern Time (US & Canada)
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Options forward straight to `CSV.parse` (e.g. `inline_csv(col_sep: ";")`). Large data sets (the multi-thousand-row variety) are better kept in a separate `.csv` you load yourself, so they can be opened in a spreadsheet editor.
|
|
85
|
+
|
|
86
|
+
`inline_csv` requires the `csv` library lazily, so it isn't a dependency unless you use it. `csv` ships with Ruby through 3.3 and is a bundled gem on 3.4+; on Ruby 3.4+ you may need to add `gem "csv"` to your Gemfile.
|
|
87
|
+
|
|
61
88
|
### Task-based shifts (targeted, one-off changes)
|
|
62
89
|
|
|
63
90
|
For targeted changes to specific records (e.g. fixing a bug for particular IDs), use `task` blocks instead:
|
|
@@ -86,6 +113,15 @@ end
|
|
|
86
113
|
|
|
87
114
|
Task blocks run in the context of the shift instance, so they have access to private helper methods, `dry_run?`, `log`, `skip!`, `find_exactly!`, and any other instance methods you define. Use private methods to DRY up shared lookups across tasks.
|
|
88
115
|
|
|
116
|
+
When a task is just a single helper [axn](https://github.com/teamshares/axn), pass the class instead of a block — its keyword args are forwarded to `.call!`, so a failure is reported instead of silently swallowed:
|
|
117
|
+
|
|
118
|
+
```ruby
|
|
119
|
+
task "Recalculate company totals", RecalculateTotals, company_id: 123
|
|
120
|
+
# equivalent to: task("Recalculate company totals") { RecalculateTotals.call!(company_id: 123) }
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
The kwargs are evaluated at class-load time, so this form is for static values; use the block form when you need runtime or instance state.
|
|
124
|
+
|
|
89
125
|
Task blocks:
|
|
90
126
|
|
|
91
127
|
- Run in sequence within the same lifecycle (transaction, dry run protection, summary)
|
|
@@ -105,6 +141,8 @@ Shifts run in **dry run** mode by default. DB changes are always rolled back in
|
|
|
105
141
|
- **Commit**: `COMMIT=1 rake data:shift:backfill_foo`
|
|
106
142
|
- (`COMMIT=true` or `DRY_RUN=false` also commit)
|
|
107
143
|
|
|
144
|
+
`COMMIT` and `DRY_RUN` are parsed as booleans — `1`, `true`, `t`, `yes`, `y`, `on` are truthy; `0`, `false`, `f`, `no`, `n`, `off` are falsey (case- and whitespace-insensitive). `COMMIT=<truthy>` commits; otherwise `DRY_RUN` decides, defaulting to a dry run — so `DRY_RUN=1` is a **dry run**, not a commit. An unrecognized value (e.g. `COMMIT=please`) raises rather than guessing a side.
|
|
145
|
+
|
|
108
146
|
### Automatic side-effect guards (dry run)
|
|
109
147
|
|
|
110
148
|
In **dry run** mode, DataShifter automatically blocks or fakes these side effects so unguarded code is less likely to hit the network or send mail/jobs:
|
|
@@ -310,11 +348,14 @@ Skip reasons are grouped: the summary shows the top 10 reasons by count (e.g. `"
|
|
|
310
348
|
|
|
311
349
|
```ruby
|
|
312
350
|
class SomeShift < DataShifter::Shift
|
|
313
|
-
throttle 0.1
|
|
314
|
-
|
|
351
|
+
throttle 0.1 # sleep 0.1s after every record
|
|
352
|
+
throttle 1.second, per: 100 # sleep 1s after every 100 records
|
|
353
|
+
progress false # disable progress bar rendering
|
|
315
354
|
end
|
|
316
355
|
```
|
|
317
356
|
|
|
357
|
+
`per:` is optional. Without it, the sleep runs after every record. With `per: N`, the sleep runs after every Nth record — useful for large collections where per-record sleeping would add too much wall-clock time.
|
|
358
|
+
|
|
318
359
|
|
|
319
360
|
## Generator
|
|
320
361
|
|
|
@@ -367,7 +408,8 @@ end
|
|
|
367
408
|
## Requirements
|
|
368
409
|
|
|
369
410
|
- Ruby ≥ 3.2.1
|
|
370
|
-
- Rails (ActiveRecord, ActiveSupport, Railties) ≥ 7.
|
|
411
|
+
- Rails (ActiveRecord, ActiveSupport, Railties) ≥ 7.2
|
|
371
412
|
- `axn` (Shift classes include `Axn`)
|
|
372
413
|
- `ruby-progressbar` (for progress bars)
|
|
373
414
|
- `webmock` (for dry-run HTTP blocking; optional allowlist via `allow_external_requests [...]` / `DataShifter.config.allow_external_requests`)
|
|
415
|
+
- `csv` — only if you use `inline_csv`; required lazily, not a hard dependency (ships with Ruby through 3.3, a bundled gem on 3.4+)
|
|
@@ -7,14 +7,27 @@ module DataShifter
|
|
|
7
7
|
module Env
|
|
8
8
|
module_function
|
|
9
9
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
10
|
+
TRUTHY = %w[1 true t yes y on].freeze
|
|
11
|
+
FALSEY = %w[0 false f no n off].freeze
|
|
12
|
+
|
|
13
|
+
# COMMIT=<truthy> means commit. Otherwise DRY_RUN decides, defaulting to a dry run.
|
|
13
14
|
def dry_run?
|
|
14
|
-
if ENV["COMMIT"].present?
|
|
15
|
-
|
|
15
|
+
return !boolean!("COMMIT") if ENV["COMMIT"].present?
|
|
16
|
+
return true if ENV["DRY_RUN"].blank?
|
|
17
|
+
|
|
18
|
+
boolean!("DRY_RUN")
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# Raises on anything unrecognized rather than picking a side. The old `DRY_RUN == "true"`
|
|
22
|
+
# compare silently read `DRY_RUN=1` as "not dry" and committed the shift — a value that is
|
|
23
|
+
# truthy in every other tool has to mean dry run here or it means data loss.
|
|
24
|
+
def boolean!(var)
|
|
25
|
+
raw = ENV.fetch(var, nil)
|
|
26
|
+
case raw.to_s.strip.downcase
|
|
27
|
+
when *TRUTHY then true
|
|
28
|
+
when *FALSEY then false
|
|
16
29
|
else
|
|
17
|
-
|
|
30
|
+
raise ArgumentError, "#{var}=#{raw.inspect} is not a boolean — use one of: #{(TRUTHY + FALSEY).join(", ")}"
|
|
18
31
|
end
|
|
19
32
|
end
|
|
20
33
|
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "axn"
|
|
4
|
+
|
|
5
|
+
# Global configuration for DataShifter, declared via the Axn::Configurable DSL.
|
|
6
|
+
#
|
|
7
|
+
# Configure via:
|
|
8
|
+
# DataShifter.configure do |config|
|
|
9
|
+
# config.allow_external_requests = ["api.readonly.example.com"]
|
|
10
|
+
# config.suppress_repeated_logs = false
|
|
11
|
+
# end
|
|
12
|
+
#
|
|
13
|
+
# Or access directly:
|
|
14
|
+
# DataShifter.config.progress_enabled = false
|
|
15
|
+
#
|
|
16
|
+
# This supplies DataShifter.config / .configure / .reset_config! plus per-setting
|
|
17
|
+
# accessors and predicates. `progress_enabled` and `suppress_repeated_logs` are
|
|
18
|
+
# `overridable: true`, so individual shifts can override them per-class (see Shift).
|
|
19
|
+
#
|
|
20
|
+
# Required from both data_shifter.rb and shift.rb (idempotent, like any require)
|
|
21
|
+
# so `include DataShifter.overrides` in Shift works whichever is required first.
|
|
22
|
+
module DataShifter
|
|
23
|
+
extend Axn::Configurable
|
|
24
|
+
|
|
25
|
+
# Namespaces DataShifter's overridable settings (`suppress_repeated_logs`, `progress_enabled`) so
|
|
26
|
+
# they can't collide with axn core's own `:core`-namespaced overridable settings — both compose
|
|
27
|
+
# onto every Shift subclass, since Shift `include Axn` as well as `DataShifter.overrides`. Declared
|
|
28
|
+
# explicitly (matching Axn::Configuration's own `config_namespace :core`) even though the default
|
|
29
|
+
# (this module object) is already collision-safe; must come before any `setting`.
|
|
30
|
+
config_namespace :data_shifter
|
|
31
|
+
|
|
32
|
+
# Hosts or regexes allowed for HTTP during dry run only (combined with per-shift allow_external_requests).
|
|
33
|
+
# Has no effect in commit mode — HTTP is unrestricted when dry_run is false.
|
|
34
|
+
setting :allow_external_requests, default: []
|
|
35
|
+
|
|
36
|
+
# Whether to allow loopback HTTP (127.0.0.1, ::1, localhost) during dry runs. Default: true.
|
|
37
|
+
# Loopback is rarely "external" and is needed for tracing/metrics sidecars (Datadog agent on
|
|
38
|
+
# 8126, statsd on 8125, OTLP collector, etc.). Set to false if you want strict net blocking.
|
|
39
|
+
setting :allow_loopback_requests, default: true
|
|
40
|
+
|
|
41
|
+
# Whether to suppress repeated log messages during a shift run. Default: true.
|
|
42
|
+
# Can be overridden per shift with `suppress_repeated_logs true/false`.
|
|
43
|
+
setting :suppress_repeated_logs, default: true, overridable: true
|
|
44
|
+
|
|
45
|
+
# Maximum unique log messages to track for deduplication. Default: 1000.
|
|
46
|
+
# When exceeded, entries with count == 1 are cleared first; repeated entries are kept.
|
|
47
|
+
setting :repeated_log_cap, default: 1000
|
|
48
|
+
|
|
49
|
+
# Global default for progress bar visibility. Default: true.
|
|
50
|
+
# Per-shift `progress true/false` overrides this.
|
|
51
|
+
setting :progress_enabled, default: true, overridable: true
|
|
52
|
+
|
|
53
|
+
# Default status print interval in seconds when ENV STATUS_INTERVAL is not set. Default: nil.
|
|
54
|
+
setting :status_interval_seconds, default: nil
|
|
55
|
+
end
|
data/lib/data_shifter/shift.rb
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "axn"
|
|
4
4
|
require "active_support/isolated_execution_state"
|
|
5
|
+
require_relative "settings"
|
|
5
6
|
require_relative "internal/env"
|
|
6
7
|
require_relative "internal/output"
|
|
7
8
|
require_relative "internal/signal_handler"
|
|
@@ -49,10 +50,17 @@ require_relative "internal/colors"
|
|
|
49
50
|
module DataShifter
|
|
50
51
|
class Shift
|
|
51
52
|
include Axn
|
|
53
|
+
# Per-class overrides for `progress_enabled` and `suppress_repeated_logs`
|
|
54
|
+
# (declared `overridable: true` on the DataShifter module). Adds class-level
|
|
55
|
+
# `progress_enabled` (bare reader resolves the override, else falls back to
|
|
56
|
+
# config) / `progress_enabled_override` (and likewise for suppress_repeated_logs)
|
|
57
|
+
# accessors; the thin `progress` / `suppress_repeated_logs` aliases below
|
|
58
|
+
# preserve the historical public DSL.
|
|
59
|
+
include DataShifter.overrides
|
|
52
60
|
|
|
53
61
|
expects :dry_run, type: :boolean, default: true
|
|
54
62
|
|
|
55
|
-
|
|
63
|
+
auto_log false
|
|
56
64
|
|
|
57
65
|
around :_with_log_deduplication
|
|
58
66
|
around :_with_side_effect_guards
|
|
@@ -62,12 +70,11 @@ module DataShifter
|
|
|
62
70
|
on_error :_print_summary_for_axn_error
|
|
63
71
|
|
|
64
72
|
class_attribute :_transaction_mode, default: :single
|
|
65
|
-
class_attribute :_progress_enabled, default: nil
|
|
66
73
|
class_attribute :_description, default: nil
|
|
67
74
|
class_attribute :_task_name, default: nil
|
|
68
75
|
class_attribute :_throttle_interval, default: nil
|
|
76
|
+
class_attribute :_throttle_per, default: 1
|
|
69
77
|
class_attribute :_allow_external_requests, default: [], instance_accessor: false
|
|
70
|
-
class_attribute :_suppress_repeated_logs, default: nil, instance_accessor: false
|
|
71
78
|
class_attribute :_task_blocks, default: [], instance_accessor: false
|
|
72
79
|
|
|
73
80
|
# Internal exception used by skip! to abort the current process_record.
|
|
@@ -105,16 +112,22 @@ module DataShifter
|
|
|
105
112
|
end
|
|
106
113
|
end
|
|
107
114
|
|
|
115
|
+
# Per-shift override for progress bar visibility. Overrides DataShifter.config.progress_enabled.
|
|
116
|
+
# Thin alias over the `progress_enabled` override accessor generated by DataShifter.overrides.
|
|
117
|
+
# progress false # set override (coerced to a boolean)
|
|
118
|
+
# progress # read the raw override (nil when unset; does NOT fall back to config)
|
|
108
119
|
def progress(enabled = nil)
|
|
109
120
|
if enabled.nil?
|
|
110
|
-
|
|
121
|
+
raw = progress_enabled_override
|
|
122
|
+
Axn::Configurable::UNSET.equal?(raw) ? nil : raw
|
|
111
123
|
else
|
|
112
|
-
|
|
124
|
+
progress_enabled(!!enabled)
|
|
113
125
|
end
|
|
114
126
|
end
|
|
115
127
|
|
|
116
|
-
def throttle(interval)
|
|
128
|
+
def throttle(interval, per: 1)
|
|
117
129
|
self._throttle_interval = interval
|
|
130
|
+
self._throttle_per = per
|
|
118
131
|
end
|
|
119
132
|
|
|
120
133
|
# Allow these hosts (or regexes) for HTTP during dry run only. Combines with DataShifter.config.allow_external_requests.
|
|
@@ -125,22 +138,36 @@ module DataShifter
|
|
|
125
138
|
end
|
|
126
139
|
|
|
127
140
|
# Enable/disable log deduplication for this shift. Overrides DataShifter.config.suppress_repeated_logs.
|
|
141
|
+
# Thin alias over the `suppress_repeated_logs` override accessor generated by DataShifter.overrides
|
|
142
|
+
# (which would otherwise read the resolved value when called with no args); we always treat a call
|
|
143
|
+
# with an argument as a set, coercing to a boolean.
|
|
128
144
|
# Example: suppress_repeated_logs false
|
|
129
145
|
def suppress_repeated_logs(enabled)
|
|
130
|
-
|
|
146
|
+
super(!!enabled)
|
|
131
147
|
end
|
|
132
148
|
|
|
133
|
-
# Define a task
|
|
134
|
-
# Multiple
|
|
135
|
-
#
|
|
149
|
+
# Define a task to run instead of collection/process_record.
|
|
150
|
+
# Multiple tasks run in sequence; labels appear in errors and summary.
|
|
151
|
+
#
|
|
152
|
+
# Block form (arbitrary code, runtime/instance state available):
|
|
136
153
|
# task "Fix user A" do
|
|
137
154
|
# User.find(123).update!(...)
|
|
138
155
|
# end
|
|
139
|
-
#
|
|
140
|
-
#
|
|
141
|
-
#
|
|
142
|
-
|
|
143
|
-
|
|
156
|
+
#
|
|
157
|
+
# Axn class form (sugar for one helper axn; kwargs are static, evaluated
|
|
158
|
+
# at class-load time, and forwarded to .call! so a failure is never
|
|
159
|
+
# silently swallowed). Use the block form when you need runtime values:
|
|
160
|
+
# task "Recalculate totals", RecalculateTotals, company_id: 123
|
|
161
|
+
# # => RecalculateTotals.call!(company_id: 123)
|
|
162
|
+
def task(label = nil, axn = nil, **axn_kwargs, &block)
|
|
163
|
+
if axn
|
|
164
|
+
raise ArgumentError, "task accepts either an Axn class or a block, not both" if block
|
|
165
|
+
raise ArgumentError, "task expected an Axn class but got #{axn.inspect}" unless axn.is_a?(Class) && axn.include?(Axn)
|
|
166
|
+
|
|
167
|
+
block = -> { axn.call!(**axn_kwargs) }
|
|
168
|
+
elsif block.nil?
|
|
169
|
+
raise ArgumentError, "task requires a block or an Axn class"
|
|
170
|
+
end
|
|
144
171
|
|
|
145
172
|
self._task_blocks = (_task_blocks || []).dup + [{ label: label.presence, block: }]
|
|
146
173
|
end
|
|
@@ -174,6 +201,24 @@ module DataShifter
|
|
|
174
201
|
ids.map { |id| records_by_id[id] }
|
|
175
202
|
end
|
|
176
203
|
|
|
204
|
+
# Parse CSV colocated with the shift, after a `__END__` marker, so small
|
|
205
|
+
# data sets can live alongside the code. Returns the data rows as an array
|
|
206
|
+
# (CSV::Row objects when headers: true, the default — so `row["id"]` works);
|
|
207
|
+
# extra options forward straight to CSV.parse. Typically used as the
|
|
208
|
+
# collection:
|
|
209
|
+
# def collection = inline_csv
|
|
210
|
+
# def process_record(row) = User.find(row["id"]).update!(...)
|
|
211
|
+
# __END__
|
|
212
|
+
# id,...
|
|
213
|
+
#
|
|
214
|
+
# Lazily requires `csv` (a bundled gem on Ruby 3.4+); raises with a hint to
|
|
215
|
+
# add it to the Gemfile if unavailable.
|
|
216
|
+
def inline_csv(**csv_opts)
|
|
217
|
+
_require_csv!
|
|
218
|
+
parsed = CSV.parse(_inline_data_body, headers: true, **csv_opts)
|
|
219
|
+
parsed.is_a?(CSV::Table) ? parsed.each.to_a : parsed
|
|
220
|
+
end
|
|
221
|
+
|
|
177
222
|
def dry_run? = dry_run
|
|
178
223
|
|
|
179
224
|
def skip!(reason = nil)
|
|
@@ -192,7 +237,10 @@ module DataShifter
|
|
|
192
237
|
# --- Axn lifecycle hooks ---
|
|
193
238
|
|
|
194
239
|
def _with_log_deduplication(chain)
|
|
195
|
-
|
|
240
|
+
# DataShifter.resolve_override_for, not the generated `suppress_repeated_logs` reader:
|
|
241
|
+
# that name is shadowed by Shift's own class method (a required-arg setter, see above),
|
|
242
|
+
# which would raise ArgumentError rather than resolve the override + config fallback.
|
|
243
|
+
effective = DataShifter.resolve_override_for(self.class, :suppress_repeated_logs)
|
|
196
244
|
unless effective && defined?(::Rails) && ::Rails.respond_to?(:logger) && ::Rails.logger
|
|
197
245
|
chain.call
|
|
198
246
|
return
|
|
@@ -399,19 +447,29 @@ module DataShifter
|
|
|
399
447
|
end
|
|
400
448
|
|
|
401
449
|
def _iterate(enum, total)
|
|
402
|
-
|
|
450
|
+
# DataShifter.resolve_override_for: the shadow-proof framework path (see
|
|
451
|
+
# _with_log_deduplication). progress_enabled isn't currently shadowed on Shift,
|
|
452
|
+
# but resolving this way is consistent and doesn't depend on that staying true.
|
|
453
|
+
progress_on = DataShifter.resolve_override_for(self.class, :progress_enabled)
|
|
403
454
|
bar = Internal::ProgressBar.create(total:, dry_run: dry_run?, enabled: progress_on)
|
|
455
|
+
throttle_count = 0
|
|
404
456
|
if enum.respond_to?(:find_each)
|
|
405
457
|
enum.find_each do |record|
|
|
406
458
|
_process_one(record) { yield record }
|
|
407
459
|
bar&.increment
|
|
408
|
-
|
|
460
|
+
if _throttle_interval
|
|
461
|
+
throttle_count += 1
|
|
462
|
+
sleep(_throttle_interval) if (throttle_count % _throttle_per).zero?
|
|
463
|
+
end
|
|
409
464
|
end
|
|
410
465
|
else
|
|
411
466
|
enum.each do |record|
|
|
412
467
|
_process_one(record) { yield record }
|
|
413
468
|
bar&.increment
|
|
414
|
-
|
|
469
|
+
if _throttle_interval
|
|
470
|
+
throttle_count += 1
|
|
471
|
+
sleep(_throttle_interval) if (throttle_count % _throttle_per).zero?
|
|
472
|
+
end
|
|
415
473
|
end
|
|
416
474
|
end
|
|
417
475
|
end
|
|
@@ -445,6 +503,44 @@ module DataShifter
|
|
|
445
503
|
_print_progress
|
|
446
504
|
end
|
|
447
505
|
|
|
506
|
+
# `csv` is lazily required (not a hard dependency) so it isn't dragged into
|
|
507
|
+
# apps that never call inline_csv. csv ships with Ruby through 3.3 and is a
|
|
508
|
+
# bundled gem on 3.4+, so this normally just works; on 3.5+ without it in the
|
|
509
|
+
# Gemfile we surface an actionable message instead of a bare LoadError.
|
|
510
|
+
def _require_csv!
|
|
511
|
+
require "csv"
|
|
512
|
+
rescue LoadError
|
|
513
|
+
raise LoadError, 'inline_csv needs the csv library. Add `gem "csv"` to your Gemfile (csv is no longer a default gem on Ruby 3.4+).'
|
|
514
|
+
end
|
|
515
|
+
|
|
516
|
+
# The raw text after this shift file's `__END__` marker. Resolves the
|
|
517
|
+
# source file via the class constant (works however the shift is invoked,
|
|
518
|
+
# unlike Ruby's `DATA`, which is only defined for the main script). Uses
|
|
519
|
+
# Ripper (not a text-based split) to find the real `__END__` token, so a
|
|
520
|
+
# `__END__`-looking line inside a heredoc or comment isn't mistaken for it.
|
|
521
|
+
def _inline_data_body
|
|
522
|
+
@_inline_data_body ||= begin
|
|
523
|
+
class_name = self.class.name
|
|
524
|
+
source = class_name && Object.const_source_location(class_name)&.first
|
|
525
|
+
raise ArgumentError, "inline_csv requires the shift to be a named class defined in a file" unless source && File.exist?(source)
|
|
526
|
+
|
|
527
|
+
require "ripper"
|
|
528
|
+
content = File.read(source)
|
|
529
|
+
offset = 0
|
|
530
|
+
found = false
|
|
531
|
+
Ripper.lex(content).each do |(_pos, type, token, _state)|
|
|
532
|
+
offset += token.length
|
|
533
|
+
if type == :on___end__
|
|
534
|
+
found = true
|
|
535
|
+
break
|
|
536
|
+
end
|
|
537
|
+
end
|
|
538
|
+
raise ArgumentError, "inline_csv: no __END__ data section found in #{source}" unless found
|
|
539
|
+
|
|
540
|
+
content[offset..]
|
|
541
|
+
end
|
|
542
|
+
end
|
|
543
|
+
|
|
448
544
|
def _format_error(e)
|
|
449
545
|
msg = e.message.to_s
|
|
450
546
|
msg += "\n Caused by: #{e.cause.class}: #{e.cause.message}" if e.respond_to?(:cause) && e.cause
|
data/lib/data_shifter/version.rb
CHANGED
data/lib/data_shifter.rb
CHANGED
|
@@ -1,27 +1,8 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require_relative "data_shifter/version"
|
|
4
|
-
require_relative "data_shifter/
|
|
4
|
+
require_relative "data_shifter/settings"
|
|
5
5
|
require_relative "data_shifter/errors"
|
|
6
6
|
require_relative "data_shifter/internal/rake_exception_reporting"
|
|
7
7
|
require_relative "data_shifter/shift"
|
|
8
8
|
require_relative "data_shifter/railtie"
|
|
9
|
-
|
|
10
|
-
module DataShifter
|
|
11
|
-
class << self
|
|
12
|
-
# Returns the global configuration instance.
|
|
13
|
-
def config
|
|
14
|
-
@config ||= Configuration.new
|
|
15
|
-
end
|
|
16
|
-
|
|
17
|
-
# Yields the configuration for block-style setup.
|
|
18
|
-
#
|
|
19
|
-
# DataShifter.configure do |c|
|
|
20
|
-
# c.allow_external_requests = ["api.readonly.example.com"]
|
|
21
|
-
# c.suppress_repeated_logs = false
|
|
22
|
-
# end
|
|
23
|
-
def configure
|
|
24
|
-
yield config
|
|
25
|
-
end
|
|
26
|
-
end
|
|
27
|
-
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: data_shifter
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.3.
|
|
4
|
+
version: 0.3.4
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kali Donovan
|
|
@@ -15,35 +15,35 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: '7.
|
|
18
|
+
version: '7.2'
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
23
|
- - ">="
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: '7.
|
|
25
|
+
version: '7.2'
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: activesupport
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
29
29
|
requirements:
|
|
30
30
|
- - ">="
|
|
31
31
|
- !ruby/object:Gem::Version
|
|
32
|
-
version: '7.
|
|
32
|
+
version: '7.2'
|
|
33
33
|
type: :runtime
|
|
34
34
|
prerelease: false
|
|
35
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
36
36
|
requirements:
|
|
37
37
|
- - ">="
|
|
38
38
|
- !ruby/object:Gem::Version
|
|
39
|
-
version: '7.
|
|
39
|
+
version: '7.2'
|
|
40
40
|
- !ruby/object:Gem::Dependency
|
|
41
41
|
name: axn
|
|
42
42
|
requirement: !ruby/object:Gem::Requirement
|
|
43
43
|
requirements:
|
|
44
44
|
- - ">="
|
|
45
45
|
- !ruby/object:Gem::Version
|
|
46
|
-
version: 0.1.0.pre.alpha.
|
|
46
|
+
version: 0.1.0.pre.alpha.5
|
|
47
47
|
- - "<"
|
|
48
48
|
- !ruby/object:Gem::Version
|
|
49
49
|
version: 0.2.0
|
|
@@ -53,7 +53,7 @@ dependencies:
|
|
|
53
53
|
requirements:
|
|
54
54
|
- - ">="
|
|
55
55
|
- !ruby/object:Gem::Version
|
|
56
|
-
version: 0.1.0.pre.alpha.
|
|
56
|
+
version: 0.1.0.pre.alpha.5
|
|
57
57
|
- - "<"
|
|
58
58
|
- !ruby/object:Gem::Version
|
|
59
59
|
version: 0.2.0
|
|
@@ -63,14 +63,14 @@ dependencies:
|
|
|
63
63
|
requirements:
|
|
64
64
|
- - ">="
|
|
65
65
|
- !ruby/object:Gem::Version
|
|
66
|
-
version: '7.
|
|
66
|
+
version: '7.2'
|
|
67
67
|
type: :runtime
|
|
68
68
|
prerelease: false
|
|
69
69
|
version_requirements: !ruby/object:Gem::Requirement
|
|
70
70
|
requirements:
|
|
71
71
|
- - ">="
|
|
72
72
|
- !ruby/object:Gem::Version
|
|
73
|
-
version: '7.
|
|
73
|
+
version: '7.2'
|
|
74
74
|
- !ruby/object:Gem::Dependency
|
|
75
75
|
name: ruby-progressbar
|
|
76
76
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -107,14 +107,10 @@ executables: []
|
|
|
107
107
|
extensions: []
|
|
108
108
|
extra_rdoc_files: []
|
|
109
109
|
files:
|
|
110
|
-
- ".husky/pre-commit"
|
|
111
|
-
- ".lintstagedrc"
|
|
112
110
|
- CHANGELOG.md
|
|
113
111
|
- LICENSE.txt
|
|
114
112
|
- README.md
|
|
115
|
-
- Rakefile
|
|
116
113
|
- lib/data_shifter.rb
|
|
117
|
-
- lib/data_shifter/configuration.rb
|
|
118
114
|
- lib/data_shifter/errors.rb
|
|
119
115
|
- lib/data_shifter/internal/colors.rb
|
|
120
116
|
- lib/data_shifter/internal/env.rb
|
|
@@ -127,6 +123,7 @@ files:
|
|
|
127
123
|
- lib/data_shifter/internal/side_effect_guards.rb
|
|
128
124
|
- lib/data_shifter/internal/signal_handler.rb
|
|
129
125
|
- lib/data_shifter/railtie.rb
|
|
126
|
+
- lib/data_shifter/settings.rb
|
|
130
127
|
- lib/data_shifter/shift.rb
|
|
131
128
|
- lib/data_shifter/spec_helper.rb
|
|
132
129
|
- lib/data_shifter/version.rb
|
data/.husky/pre-commit
DELETED
|
@@ -1 +0,0 @@
|
|
|
1
|
-
npx lint-staged
|
data/.lintstagedrc
DELETED
data/Rakefile
DELETED
|
@@ -1,16 +0,0 @@
|
|
|
1
|
-
# frozen_string_literal: true
|
|
2
|
-
|
|
3
|
-
require "bundler/gem_tasks"
|
|
4
|
-
require "rubocop/rake_task"
|
|
5
|
-
|
|
6
|
-
task :spec do
|
|
7
|
-
sh "bundle exec rspec"
|
|
8
|
-
end
|
|
9
|
-
|
|
10
|
-
RuboCop::RakeTask.new
|
|
11
|
-
|
|
12
|
-
task default: %i[spec rubocop]
|
|
13
|
-
|
|
14
|
-
# Ensure specs and rubocop pass before release (must run first; enhance appends)
|
|
15
|
-
release_task = Rake::Task["release"]
|
|
16
|
-
release_task.prerequisites.unshift(:default)
|
|
@@ -1,48 +0,0 @@
|
|
|
1
|
-
# frozen_string_literal: true
|
|
2
|
-
|
|
3
|
-
module DataShifter
|
|
4
|
-
# Global configuration for DataShifter.
|
|
5
|
-
#
|
|
6
|
-
# Configure via:
|
|
7
|
-
# DataShifter.configure do |config|
|
|
8
|
-
# config.allow_external_requests = ["api.readonly.example.com"]
|
|
9
|
-
# config.suppress_repeated_logs = true
|
|
10
|
-
# end
|
|
11
|
-
#
|
|
12
|
-
# Or access directly:
|
|
13
|
-
# DataShifter.config.progress_enabled = false
|
|
14
|
-
class Configuration
|
|
15
|
-
# Hosts or regexes allowed for HTTP during dry run only (combined with per-shift allow_external_requests).
|
|
16
|
-
# Has no effect in commit mode — HTTP is unrestricted when dry_run is false.
|
|
17
|
-
attr_accessor :allow_external_requests
|
|
18
|
-
|
|
19
|
-
# Whether to allow loopback HTTP (127.0.0.1, ::1, localhost) during dry runs. Default: true.
|
|
20
|
-
# Loopback is rarely "external" and is needed for tracing/metrics sidecars (Datadog agent on
|
|
21
|
-
# 8126, statsd on 8125, OTLP collector, etc.). Set to false if you want strict net blocking.
|
|
22
|
-
attr_accessor :allow_loopback_requests
|
|
23
|
-
|
|
24
|
-
# Whether to suppress repeated log messages during a shift run. Default: true.
|
|
25
|
-
# Can be overridden per shift with `suppress_repeated_logs true/false`.
|
|
26
|
-
attr_accessor :suppress_repeated_logs
|
|
27
|
-
|
|
28
|
-
# Maximum unique log messages to track for deduplication. Default: 1000.
|
|
29
|
-
# When exceeded, entries with count == 1 are cleared first; repeated entries are kept.
|
|
30
|
-
attr_accessor :repeated_log_cap
|
|
31
|
-
|
|
32
|
-
# Global default for progress bar visibility. Default: true.
|
|
33
|
-
# Per-shift `progress true/false` overrides this.
|
|
34
|
-
attr_accessor :progress_enabled
|
|
35
|
-
|
|
36
|
-
# Default status print interval in seconds when ENV STATUS_INTERVAL is not set. Default: nil.
|
|
37
|
-
attr_accessor :status_interval_seconds
|
|
38
|
-
|
|
39
|
-
def initialize
|
|
40
|
-
@allow_external_requests = []
|
|
41
|
-
@allow_loopback_requests = true
|
|
42
|
-
@suppress_repeated_logs = true
|
|
43
|
-
@repeated_log_cap = 1000
|
|
44
|
-
@progress_enabled = true
|
|
45
|
-
@status_interval_seconds = nil
|
|
46
|
-
end
|
|
47
|
-
end
|
|
48
|
-
end
|