logidze 0.12.0 → 1.0.0.rc1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (58) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +29 -5
  3. data/LICENSE.txt +1 -1
  4. data/README.md +263 -103
  5. data/lib/generators/logidze/fx_helper.rb +17 -0
  6. data/lib/generators/logidze/inject_sql.rb +18 -0
  7. data/lib/generators/logidze/install/USAGE +6 -1
  8. data/lib/generators/logidze/install/functions/logidze_compact_history.sql +38 -0
  9. data/lib/generators/logidze/install/functions/logidze_filter_keys.sql +27 -0
  10. data/lib/generators/logidze/install/functions/logidze_logger.sql +150 -0
  11. data/lib/generators/logidze/install/functions/logidze_snapshot.sql +24 -0
  12. data/lib/generators/logidze/install/functions/logidze_version.sql +20 -0
  13. data/lib/generators/logidze/install/install_generator.rb +58 -1
  14. data/lib/generators/logidze/install/templates/hstore.rb.erb +1 -1
  15. data/lib/generators/logidze/install/templates/migration.rb.erb +19 -232
  16. data/lib/generators/logidze/install/templates/migration_fx.rb.erb +41 -0
  17. data/lib/generators/logidze/model/model_generator.rb +49 -13
  18. data/lib/generators/logidze/model/templates/migration.rb.erb +57 -36
  19. data/lib/generators/logidze/model/triggers/logidze.sql +6 -0
  20. data/lib/logidze.rb +27 -14
  21. data/lib/logidze/history.rb +1 -10
  22. data/lib/logidze/ignore_log_data.rb +1 -4
  23. data/lib/logidze/model.rb +48 -35
  24. data/lib/logidze/version.rb +1 -1
  25. metadata +48 -73
  26. data/.gitattributes +0 -3
  27. data/.github/ISSUE_TEMPLATE.md +0 -20
  28. data/.github/PULL_REQUEST_TEMPLATE.md +0 -29
  29. data/.gitignore +0 -40
  30. data/.rubocop.yml +0 -55
  31. data/.travis.yml +0 -46
  32. data/Gemfile +0 -15
  33. data/Rakefile +0 -28
  34. data/assets/pg_log_data_chart.png +0 -0
  35. data/bench/performance/README.md +0 -109
  36. data/bench/performance/diff_bench.rb +0 -38
  37. data/bench/performance/insert_bench.rb +0 -22
  38. data/bench/performance/memory_profile.rb +0 -56
  39. data/bench/performance/setup.rb +0 -315
  40. data/bench/performance/update_bench.rb +0 -38
  41. data/bench/triggers/Makefile +0 -56
  42. data/bench/triggers/Readme.md +0 -58
  43. data/bench/triggers/bench.sql +0 -6
  44. data/bench/triggers/hstore_trigger_setup.sql +0 -38
  45. data/bench/triggers/jsonb_minus_2_setup.sql +0 -47
  46. data/bench/triggers/jsonb_minus_setup.sql +0 -49
  47. data/bench/triggers/keys2_trigger_setup.sql +0 -44
  48. data/bench/triggers/keys_trigger_setup.sql +0 -50
  49. data/bin/console +0 -8
  50. data/bin/setup +0 -9
  51. data/gemfiles/rails42.gemfile +0 -6
  52. data/gemfiles/rails5.gemfile +0 -6
  53. data/gemfiles/rails52.gemfile +0 -6
  54. data/gemfiles/rails6.gemfile +0 -6
  55. data/gemfiles/railsmaster.gemfile +0 -7
  56. data/lib/logidze/ignore_log_data/ignored_columns.rb +0 -46
  57. data/lib/logidze/migration.rb +0 -20
  58. data/logidze.gemspec +0 -41
data/Gemfile DELETED
@@ -1,15 +0,0 @@
1
- # frozen_string_literal: true
2
- source 'https://rubygems.org'
3
-
4
- # Specify your gem's dependencies in logidze.gemspec
5
- gemspec
6
-
7
- gem "pry-byebug", platform: :mri
8
-
9
- local_gemfile = "#{File.dirname(__FILE__)}/Gemfile.local"
10
-
11
- if File.exist?(local_gemfile)
12
- eval(File.read(local_gemfile)) # rubocop:disable Lint/Eval
13
- else
14
- gem 'activerecord', '~> 6.0'
15
- end
data/Rakefile DELETED
@@ -1,28 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require "rspec/core/rake_task"
4
- require "rubocop/rake_task"
5
-
6
- RuboCop::RakeTask.new
7
- RSpec::Core::RakeTask.new(:spec)
8
-
9
- namespace :dummy do
10
- require_relative "spec/dummy/config/application"
11
- Dummy::Application.load_tasks
12
- end
13
-
14
- task(:spec).clear
15
- desc "Run specs other than spec/acceptance"
16
- RSpec::Core::RakeTask.new("spec") do |task|
17
- task.exclude_pattern = "spec/acceptance/**/*_spec.rb"
18
- task.verbose = false
19
- end
20
-
21
- desc "Run acceptance specs in spec/acceptance"
22
- RSpec::Core::RakeTask.new("spec:acceptance") do |task|
23
- task.pattern = "spec/acceptance/**/*_spec.rb"
24
- task.verbose = false
25
- end
26
-
27
- desc "Run the specs and acceptance tests"
28
- task default: %w[rubocop spec spec:acceptance]
Binary file
@@ -1,109 +0,0 @@
1
- # Performance benchmarks: PaperTail vs. Logidze
2
-
3
- We want to compare Logidze with the most popular versioning library for Rails – PaperTrail.
4
-
5
-
6
- ## Insert ([source](insert_bench.rb))
7
-
8
- ```
9
- PaperTrail INSERT 213.148 (± 8.9%) i/s - 1.060k in 5.018504s
10
- Logidze INSERT 613.387 (±16.3%) i/s - 2.970k in 5.036127s
11
- ```
12
-
13
-
14
- ## Update ([source](update_bench.rb))
15
-
16
- When changeset has 2 fields:
17
-
18
- ```
19
- PaperTrail UPDATE #1 256.651 (±26.5%) i/s - 1.206k in 5.002300s
20
- Logidze UPDATE #1 356.932 (±12.6%) i/s - 1.764k in 5.030560s
21
- ```
22
-
23
- When changeset has 5 fields:
24
-
25
- ```
26
- PaperTrail UPDATE #2 246.281 (±24.0%) i/s - 1.168k in 5.008234s
27
- Logidze UPDATE #2 331.942 (±16.6%) i/s - 1.593k in 5.028135s
28
- ```
29
-
30
- ## Getting diff ([source](diff_bench.rb))
31
-
32
- PaperTrail doesn't have built-in method to calculate diff between not adjacent versions.
33
- We add `diff_from(ts)` and `diff_from_joined(ts)` (which uses SQL JOIN) methods to calculate diff from specified version using changesets.
34
-
35
- When each record has 10 versions:
36
-
37
- ```
38
- PT DIFF 20.874 (± 4.8%) i/s - 106.000 in 5.091402s
39
- PT (join) DIFF 20.619 (± 4.8%) i/s - 104.000 in 5.070160s
40
- Logidze DIFF 109.482 (±24.7%) i/s - 500.000 in 5.103534s
41
- ```
42
-
43
- When each record has 100 versions:
44
-
45
- ```
46
- PT DIFF 2.998 (± 0.0%) i/s - 15.000 in 5.019494s
47
- PT (join) DIFF 3.193 (± 0.0%) i/s - 16.000 in 5.030155s
48
- Logidze DIFF 19.627 (±25.5%) i/s - 88.000 in 5.035555s
49
- ```
50
-
51
- And, finally, when each record has 1000 versions:
52
-
53
- ```
54
- PT DIFF 0.270 (± 0.0%) i/s - 17.000 in 63.038374s
55
- PT (join) DIFF 0.235 (± 0.0%) i/s - 14.000 in 60.350886s
56
- Logidze DIFF 2.022 (± 0.0%) i/s - 120.000 in 60.142965s
57
- ```
58
-
59
- ## Select memory usage ([source](memory_profile.rb))
60
-
61
- Logidze loads more data (because it stores log in-place). But how much more?
62
- We consider two cases for PaperTrail: when we want to calculate diff (and thus loading versions) and when we don't need any history related data.
63
-
64
- When each record has 10 versions:
65
-
66
- ```
67
- PT records
68
- Total Allocated: 27.8 KB
69
- Total Retained: 16.59 KB
70
- Retained memory (per record): 2.14 KB
71
-
72
- PT with versions
73
- Total Allocated: 228.01 KB
74
- Total Retained: 170.78 KB
75
- Retained memory (per record): 143.13 KB
76
-
77
- Logidze records
78
- Total Allocated: 46.45 KB
79
- Total Retained: 34.73 KB
80
- Retained memory (per record): 4.11 KB
81
- ```
82
-
83
- When each record has 100 versions:
84
-
85
- ```
86
- PT with versions
87
- Total Allocated: 1.92 MB
88
- Total Retained: 1.56 MB
89
- Retained memory (per record): 1.53 MB
90
-
91
- Logidze records
92
- Total Allocated: 162.48 KB
93
- Total Retained: 150.76 KB
94
- Retained memory (per record): 15.4 KB
95
- ```
96
-
97
- When each record has 1000 versions:
98
-
99
- ```
100
- PT with versions
101
- Total Allocated: 18.23 MB
102
- Total Retained: 14.86 MB
103
- Retained memory (per record): 14.83 MB
104
-
105
- Logidze records
106
- Total Allocated: 1.32 MB
107
- Total Retained: 1.31 MB
108
- Retained memory (per record): 131.59 KB
109
- ```
@@ -1,38 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require "benchmark/ips"
4
- require "./setup"
5
-
6
- # How many records do you want?
7
- N = (ENV["N"] || "100").to_i
8
-
9
- # How many version each record has?
10
- V = (ENV["V"] || "10").to_i
11
-
12
- # Benchmark run time
13
- BM_TIME = (ENV["BM_TIME"] || 5).to_i
14
-
15
- BM_WARMUP = [(BM_TIME / 10), 2].max
16
-
17
- LogidzeBench.cleanup
18
- LogidzeBench.populate(N)
19
-
20
- ts1 = LogidzeBench.generate_versions(V / 2)
21
-
22
- LogidzeBench.generate_versions(V / 2)
23
-
24
- Benchmark.ips do |x|
25
- x.config(time: BM_TIME, warmup: BM_WARMUP)
26
-
27
- x.report("PT DIFF") do
28
- User.random(N / 2).diff_from(ts1)
29
- end
30
-
31
- x.report("PT (join) DIFF") do
32
- User.random(N / 2).diff_from_joined(ts1)
33
- end
34
-
35
- x.report("Logidze DIFF") do
36
- LogidzeUser.random(N / 2).diff_from(ts1)
37
- end
38
- end
@@ -1,22 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require "benchmark/ips"
4
- require "./setup"
5
-
6
- params = {
7
- email: Faker::Internet.email,
8
- position: Faker::Number.number(3),
9
- name: Faker::Name.name,
10
- age: Faker::Number.number(2),
11
- bio: Faker::Lorem.paragraph
12
- }
13
-
14
- Benchmark.ips do |x|
15
- x.report("PaperTrail INSERT") do
16
- User.create!(params)
17
- end
18
-
19
- x.report("Logidze INSERT") do
20
- LogidzeUser.create!(params)
21
- end
22
- end
@@ -1,56 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require "./setup"
4
- require "active_support/core_ext"
5
- require "memory_profiler"
6
-
7
- # How many records do you want?
8
- N = (ENV["N"] || "10").to_i
9
-
10
- # How many version each record has?
11
- V = (ENV["V"] || "10").to_i
12
-
13
- LogidzeBench.cleanup
14
- LogidzeBench.populate(N)
15
- LogidzeBench.generate_versions(V)
16
-
17
- module MemoryReport
18
- KILO_BYTE = 1024
19
- MEGA_BYTE = 1024 * 1024
20
-
21
- module_function
22
-
23
- def call(msg, relation)
24
- buffer = nil
25
- delta = N / 10
26
- r0 = MemoryProfiler.report do
27
- buffer = relation.random(N - delta).to_a
28
- end
29
-
30
- buffer = nil
31
- r1 = MemoryProfiler.report do
32
- buffer = relation.to_a
33
- end
34
-
35
- $stdout.puts msg
36
- $stdout.puts "Total Allocated:\t\t\t\t#{to_human_size(r1.total_allocated_memsize)}"
37
- $stdout.puts "Total Retained:\t\t\t\t\t#{to_human_size(r1.total_retained_memsize)}"
38
- $stdout.puts "Retained_memsize memory (per record):\t\t#{to_human_size((r1.total_retained_memsize - r0.total_retained_memsize) / delta)}"
39
- end
40
-
41
- module_function
42
-
43
- def to_human_size(size)
44
- if size > MEGA_BYTE
45
- "#{(size.to_f / MEGA_BYTE).round(2)} MB"
46
- elsif size > KILO_BYTE
47
- "#{(size.to_f / KILO_BYTE).round(2)} KB"
48
- else
49
- "#{size} B"
50
- end
51
- end
52
- end
53
-
54
- MemoryReport.call("PT records", User.all)
55
- MemoryReport.call("PT with versions", User.joins(:versions).all)
56
- MemoryReport.call("Logidze records", LogidzeUser.all)
@@ -1,315 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- begin
4
- require "bundler/inline"
5
- rescue LoadError => e
6
- warn "Bundler version 1.10 or later is required. Please update your Bundler"
7
- raise e
8
- end
9
-
10
- gemfile(true) do
11
- source "https://rubygems.org"
12
- gem "activerecord", "~>4.2"
13
- gem "pg"
14
- gem "paper_trail", "~>4.2", require: false
15
- gem "pry-byebug"
16
- gem "faker"
17
- gem "benchmark-ips"
18
- gem "memory_profiler"
19
- end
20
-
21
- DB_NAME = ENV["DB_NAME"] || "logidze_query_bench"
22
-
23
- begin
24
- system("createdb #{DB_NAME}")
25
- rescue
26
- $stdout.puts "DB already exists"
27
- end
28
-
29
- $LOAD_PATH.unshift File.expand_path("../../../lib", __FILE__)
30
-
31
- require "active_record"
32
- require "logger"
33
- require "logidze"
34
-
35
- ActiveRecord::Base.send :include, Logidze::HasLogidze
36
-
37
- ActiveRecord::Base.establish_connection(adapter: "postgresql", database: DB_NAME)
38
-
39
- at_exit do
40
- ActiveRecord::Base.connection.disconnect!
41
- end
42
-
43
- require "paper_trail"
44
-
45
- module LogidzeBench
46
- module_function
47
-
48
- def setup_db
49
- ActiveRecord::Schema.define do
50
- # PaperTrail setup
51
- create_table :versions, force: true do |t|
52
- t.string :item_type, null: false
53
- t.integer :item_id, null: false
54
- t.string :event, null: false
55
- t.string :whodunnit
56
- t.text :object
57
- t.jsonb :object_changes
58
-
59
- t.datetime :created_at
60
- end
61
-
62
- add_index :versions, [:item_type, :item_id]
63
-
64
- # Logidze setup
65
- enable_extension :hstore
66
-
67
- execute <<~SQL
68
- DO $$
69
- BEGIN
70
- EXECUTE 'ALTER DATABASE ' || current_database() || ' SET logidze.disabled TO off';
71
- END;
72
- $$
73
- LANGUAGE plpgsql;
74
- SQL
75
-
76
- execute <<~SQL
77
- CREATE OR REPLACE FUNCTION logidze_logger() RETURNS TRIGGER AS $body$
78
- DECLARE
79
- changes jsonb;
80
- new_v integer;
81
- ts bigint;
82
- size integer;
83
- history_limit integer;
84
- current_version integer;
85
- merged jsonb;
86
- iterator integer;
87
- item record;
88
- BEGIN
89
- ts := (extract(epoch from now()) * 1000)::bigint;
90
-
91
- IF TG_OP = 'INSERT' THEN
92
- changes := to_jsonb(NEW.*) - 'log_data';
93
- new_v := 1;
94
-
95
- NEW.log_data := json_build_object(
96
- 'v',
97
- 1,
98
- 'h',
99
- jsonb_build_array(
100
- jsonb_build_object(
101
- 'ts',
102
- ts,
103
- 'v',
104
- new_v,
105
- 'c',
106
- changes
107
- )
108
- )
109
- );
110
- ELSIF TG_OP = 'UPDATE' THEN
111
- history_limit := TG_ARGV[0];
112
- current_version := (NEW.log_data->>'v')::int;
113
-
114
- IF NEW = OLD THEN
115
- RETURN NEW;
116
- END IF;
117
-
118
- IF current_version < (NEW.log_data#>>'{h,-1,v}')::int THEN
119
- iterator := 0;
120
- FOR item in SELECT * FROM jsonb_array_elements(NEW.log_data->'h')
121
- LOOP
122
- IF (item.value->>'v')::int > current_version THEN
123
- NEW.log_data := jsonb_set(
124
- NEW.log_data,
125
- '{h}',
126
- (NEW.log_data->'h') - iterator
127
- );
128
- END IF;
129
- iterator := iterator + 1;
130
- END LOOP;
131
- END IF;
132
-
133
- changes := hstore_to_jsonb_loose(
134
- hstore(NEW.*) - hstore(OLD.*)
135
- ) - 'log_data';
136
-
137
- new_v := (NEW.log_data#>>'{h,-1,v}')::int + 1;
138
-
139
- size := jsonb_array_length(NEW.log_data->'h');
140
-
141
- NEW.log_data := jsonb_set(
142
- NEW.log_data,
143
- ARRAY['h', size::text],
144
- jsonb_build_object(
145
- 'ts',
146
- ts,
147
- 'v',
148
- new_v,
149
- 'c',
150
- changes
151
- ),
152
- true
153
- );
154
-
155
- NEW.log_data := jsonb_set(
156
- NEW.log_data,
157
- '{v}',
158
- to_jsonb(new_v)
159
- );
160
-
161
- IF history_limit IS NOT NULL AND history_limit = size THEN
162
- merged := jsonb_build_object(
163
- 'ts',
164
- NEW.log_data#>'{h,1,ts}',
165
- 'v',
166
- NEW.log_data#>'{h,1,v}',
167
- 'c',
168
- (NEW.log_data#>'{h,0,c}') || (NEW.log_data#>'{h,1,c}')
169
- );
170
-
171
- NEW.log_data := jsonb_set(
172
- NEW.log_data,
173
- '{h}',
174
- jsonb_set(
175
- NEW.log_data->'h',
176
- '{1}',
177
- merged
178
- ) - 0
179
- );
180
- END IF;
181
- END IF;
182
-
183
- return NEW;
184
- END;
185
- $body$
186
- LANGUAGE plpgsql;
187
- SQL
188
-
189
- create_table :users, force: true do |t|
190
- t.string :email
191
- t.integer :position
192
- t.string :name
193
- t.text :bio
194
- t.integer :age
195
- t.timestamps
196
- end
197
-
198
- create_table :logidze_users, force: true do |t|
199
- t.string :email
200
- t.integer :position
201
- t.string :name
202
- t.text :bio
203
- t.integer :age
204
- t.jsonb :log_data, default: "{}", null: false
205
- t.timestamps
206
- end
207
-
208
- execute <<~SQL
209
- CREATE TRIGGER logidze_on_logidze_users
210
- BEFORE UPDATE OR INSERT ON logidze_users FOR EACH ROW
211
- WHEN (current_setting('logidze.disabled') <> 'on')
212
- EXECUTE PROCEDURE logidze_logger();
213
- SQL
214
- end
215
- end
216
-
217
- module_function
218
-
219
- def populate(n = 1_000)
220
- n.times do
221
- params = fake_params
222
- User.create!(params)
223
- LogidzeUser.create!(params)
224
- end
225
- end
226
-
227
- module_function
228
-
229
- def cleanup
230
- LogidzeUser.delete_all
231
- User.delete_all
232
- PaperTrail::Version.delete_all
233
- end
234
-
235
- module_function
236
-
237
- def generate_versions(num = 1)
238
- num.times do
239
- User.find_each do |u|
240
- u.update!(fake_params(sample: true))
241
- end
242
-
243
- LogidzeUser.find_each do |u|
244
- u.update!(fake_params(sample: true))
245
- end
246
-
247
- # make at least 1 second between versions
248
- sleep 1
249
- end
250
- Time.now
251
- end
252
-
253
- module_function
254
-
255
- def fake_params(sample: false)
256
- params = {
257
- email: Faker::Internet.email,
258
- position: Faker::Number.number(3),
259
- name: Faker::Name.name,
260
- age: Faker::Number.number(2),
261
- bio: Faker::Lorem.paragraph
262
- }
263
-
264
- return params.slice(%i[email position name age bio].sample) if sample
265
- params
266
- end
267
- end
268
-
269
- module ARandom
270
- def random(num = 1)
271
- rel = order("random()")
272
- num == 1 ? rel.first : rel.limit(num)
273
- end
274
- end
275
-
276
- class User < ActiveRecord::Base
277
- extend ARandom
278
- has_paper_trail
279
-
280
- def self.diff_from(ts)
281
- includes(:versions).map { |u| {"id" => u.id, "changes" => u.diff_from(ts)} }
282
- end
283
-
284
- def self.diff_from_joined(ts)
285
- eager_load(:versions).map { |u| {"id" => u.id, "changes" => u.diff_from(ts)} }
286
- end
287
-
288
- def diff_from(ts)
289
- changes = {}
290
- versions.each do |v|
291
- next if v.created_at < ts
292
- merge_changeset(changes, v.changeset)
293
- end
294
- changes
295
- end
296
-
297
- private
298
-
299
- def merge_changeset(acc, data)
300
- data.each do |k, v|
301
- unless acc.key?(k)
302
- acc[k] = {"old" => v[0]}
303
- end
304
- acc[k]["new"] = v[1]
305
- end
306
- end
307
- end
308
-
309
- class LogidzeUser < ActiveRecord::Base
310
- extend ARandom
311
- has_logidze
312
- end
313
-
314
- # Run migration only if neccessary
315
- LogidzeBench.setup_db if ENV["FORCE"].present? || !ActiveRecord::Base.connection.tables.include?("logidze_users")