live_fixtures 0.2.0 → 2.0.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 65c289e9334bcc64e494914412b0009ba331df5d
4
- data.tar.gz: 1c4e83eb54bc2e270b425e04626fe24d15b0f327
2
+ SHA256:
3
+ metadata.gz: 4555eba9da64e35ef61d763132a1d2e5a280c785d0b2583fcb9055e6800eae74
4
+ data.tar.gz: fb54c037ff212fecb9bdbe2fd0c961df6d0acba9a0ed6dab7579c5e487656b0f
5
5
  SHA512:
6
- metadata.gz: 1ce276cc748f25dd873cd6685da27f96893cba3a1fe254b3dfaf135f46c2ed2db485b9710ad9d45f53f10faa2759a54d419b00fee50c8e9efd147ae83b7a3bae
7
- data.tar.gz: 16dafe289086b77e9177766fe059451258c1647d84b1c03249c83f965a176930be4d9d9578ab72ac50a2388040bd37ab24751f551398ddeeb3ccc9f48e1a4ae5
6
+ metadata.gz: 6f0af7f78d0638a22f2dc607b6a21b0cf24b95497fe974c4f0c1b1d7c4b895048deac44ec55576391ccf67bc297e8aa318e841ed1ef1669eb286b8cdeba3f932
7
+ data.tar.gz: 12a8488d4e93c45462e7362926f057571f54f4afe2640ff94dc422b8c3bbc9a4ff0a6b4a064962e15ca13b75c2abb60e1f016495189a28e3d14b627504152c32
@@ -2,7 +2,53 @@
2
2
  All notable changes to this project will be documented in this file.
3
3
  This project adheres to [Semantic Versioning](http://semver.org/).
4
4
 
5
- ## [0.2.0] - (Unreleased)
5
+ ## [2.0.0] - 2020-11-11
6
+ ### Breaking changes
7
+ This is a breaking change because LiveFixtures::Import.new now needs to receive the class_names Hash to be able to correctly compute the insert_order in case there are some unconventional class names associations. And the class_names argument is removed from import_all. But this is the only change.
8
+
9
+ ### Added
10
+ - compute insert_order automatically (#33)
11
+
12
+ ## [1.0.1] - 2019-04-10
13
+ ### Fixed
14
+ - fixed incompatibility with mysql
15
+
16
+ ### Added
17
+ - mysql regression test, confirmation that this gem doesn't work with psotgres
18
+
19
+ ## [1.0.0] - 2019-02-15
20
+ ### Breaking changes
21
+ - drop support for rails 4.2, ruby < 2.3
22
+
23
+ ### Fixed
24
+ - support for rails 5
25
+
26
+ ### Added
27
+ - None
28
+
29
+ ## [0.3.1] - 2018-03-28
30
+ ### Breaking changes
31
+ - None
32
+
33
+ ### Fixed
34
+ - None
35
+
36
+ ### Added
37
+ - It is now possible to export an attribute named "id" when it is included among the [additional attributes](https://github.com/NoRedInk/live_fixtures/tree/3868aaddbeb1c0174261673855610c4f8d9e7842#additional-attributes). #25
38
+
39
+ ## [0.3.0] - 2017-08-10
40
+ ### Breaking changes
41
+ - Imports now raise an error when unable to find a referenced model.
42
+ To avoid this behavior, pass the option `skip_missing_refs: true`. #24
43
+
44
+ ### Fixed
45
+ - For models with serialized attributes (for storing a ruby object or json blob in a single column, for example), live_fixtures will now use the coder specified in the model definition to dump the value to a string, rather than serializing it to yaml. #10, #11
46
+
47
+ ### Added
48
+ - Enhanced documentation #1, #12
49
+ - Options to suppress progress bar output and import errors. #23
50
+
51
+ ## [0.2.0] - 2017-05-09
6
52
  ### Breaking change
7
53
  - live_fixtures now depends on activerecord ~> 4.2
8
54
 
data/README.md CHANGED
@@ -1,16 +1,26 @@
1
1
  # LiveFixtures
2
2
 
3
- Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/live_fixtures`. To experiment with that code, run `bin/console` for an interactive prompt.
3
+ Like ActiveRecord::Fixtures, but for production.
4
4
 
5
- TODO: Delete this and the text above, and describe your gem
5
+ > A test fixture is a fixed state of a set of objects used as a baseline for running tests.
6
+ > The purpose of a test fixture is to ensure that there is a well known and fixed environment in which tests are run so that results are repeatable.
7
+ >
8
+ > [https://github.com/junit-team/junit4/wiki/test-fixtures](https://github.com/junit-team/junit4/wiki/test-fixtures)
9
+
10
+ [ActiveRecord::Fixtures](http://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html) provide a powerful way to populate a database with test fixtures, but its strategy for handling primary keys and associations is not intended for use with a production db.
11
+
12
+ LiveFixtures uses a different strategy that means it is safer to use in a live environment.
13
+
14
+ For more information, see [the motivation section below](#motivation).
15
+
16
+ ## Compatibility
17
+ LiveFixtures is tested against Sqlite3 & mysql. It is known to be incompatible with postgres.
6
18
 
7
19
  ## Installation
8
20
 
9
21
  Add this line to your application's Gemfile:
10
22
 
11
- ```ruby
12
- gem 'live_fixtures'
13
- ```
23
+ gem 'live_fixtures'
14
24
 
15
25
  And then execute:
16
26
 
@@ -22,7 +32,256 @@ Or install it yourself as:
22
32
 
23
33
  ## Usage
24
34
 
25
- TODO: Write usage instructions here
35
+ This gem provides functionality for both the export and the import of fixtures.
36
+
37
+ While they work nicely together, it is also possible to import manually generated fixtures.
38
+
39
+ ### Exporting
40
+
41
+ The `LiveFixtures::Export` module is meant to be included into your export class.
42
+
43
+
44
+ class Export::User
45
+ include LiveFixtures::Export
46
+
47
+ def export(user_id)
48
+ set_export_dir "#{Rails.root}/data/export/user/#{user_id}/"
49
+
50
+ export_user_and_posts(user_id)
51
+ end
52
+
53
+ def export_user_and_posts(user_id)
54
+ user = User.find(user_id)
55
+ export_fixtures([user])
56
+
57
+ export_fixtures user.posts, :user do |post|
58
+ { "likes" => post.likes.count,
59
+ "unique_url" => Template.new("<%= Export::Helper.unique_url %>") }
60
+ end
61
+ end
62
+ end
63
+
64
+
65
+ 1. Call #set_export_dir to set the dir where files should be created.
66
+ If the dir does not already exist, it will be created for you.
67
+
68
+ 2. Then call #export_fixtures for each db table, which will produce
69
+ one yml file for each db table. Do *not* call export_fixtures multiple
70
+ times for the same db table - that will overwrite the file each time!
71
+
72
+ 3. You can optionally call #set_export_options, passing {show_progress: false}
73
+ if you'd like to disable the progress bar.
74
+
75
+ 4. For advanced usage, read the sections about Additional Attributes, References, and Templates.
76
+
77
+ ### Importing
78
+
79
+ The `LiveFixtures::Import` class allows you to specify the location of your fixtures and, optionally, the order in which to import them. If you don't specify an order, the order will be computed from the ActiveRecord models associations. Once you've done that, you can import them directly to your database.
80
+
81
+
82
+ module Seed::User
83
+ def self.from_fixtures(fixtures_directory)
84
+ insert_order = %w{users posts}
85
+
86
+ importer = LiveFixtures::Import.new fixtures_directory, insert_order
87
+ importer.import_all
88
+ end
89
+ end
90
+
91
+ Options may be passed when initializing an importer as follow:
92
+ - show_progress: defaults to true.
93
+ Pass false to disable the progress bar output.
94
+ - skip_missing_tables: defaults to false.
95
+ Pass true to avoid raising an error when a table listed in insert_order has
96
+ no yml file.
97
+ - skip_missing_refs: defaults to false.
98
+ Pass false to raise an error when the importer is unable to re-establish a
99
+ relation.
100
+
101
+ ## Advanced Usage
102
+
103
+ The following topics reference this schema and exporter:
104
+
105
+ class User < ActiveRecord::Base
106
+ has_many :posts
107
+ end
108
+
109
+ class Post < ActiveRecord::Base
110
+ belongs_to :user
111
+ has_and_belongs_to_many :channels
112
+ end
113
+
114
+ class Channel < ActiveRecord::Base
115
+ has_and_belongs_to_many :users
116
+ end
117
+
118
+ class YourExporter
119
+ include LiveFixtures::Export
120
+ def initialize(fixture_path)
121
+ set_export_dir fixture_path
122
+ end
123
+
124
+ def export_models(models, references = [], &additional_attributes)
125
+ export_fixtures(models, references, &additional_attributes)
126
+ end
127
+ end
128
+
129
+ ### Additional Attributes
130
+
131
+ We can use a block to add more attributes to a fixture. Each model is passed to the block as an argument, and the block should return a hash of additional arguments.
132
+
133
+ dev_ops_posts = Post.where(topic: "Dev Ops")
134
+ exporter = YourExporter.new("fixtures/")
135
+ exporter.export_models(dev_ops_posts) do |post|
136
+ { summary: PostSummarizer.summarize(post) }
137
+ end
138
+
139
+ # In our fixtures/posts.yml file
140
+ posts_1234:
141
+ ...
142
+ user_id: 5678
143
+ summary: "Dev ops is cool."
144
+
145
+ ### References
146
+
147
+ References allow fixtures to capture a model's associations, so they can be correctly re-established on import.
148
+
149
+ When we export a fixture for a post above, we'd expect to see an attribute `user_id`
150
+
151
+ post = Post.find(1234)
152
+ post.user_id = 5678
153
+ post.save!
154
+ exporter = YourExporter.new("fixtures/")
155
+ exporter.export_models([post])
156
+
157
+ # In our posts.yml file
158
+ posts_1234:
159
+ ...
160
+ user_id: 5678
161
+
162
+
163
+ If we import this post, it will still have 5678 for its user_id foreign key. This may or may not be the desired outcome.
164
+
165
+ If we pass `:user` as references, LiveFixtures will replace the foreign key with a reference, so that the association can be correctly re-established on import:
166
+
167
+ post = Post.find(1234)
168
+ post.user_id = 5678
169
+ post.save!
170
+ exporter = YourExporter.new("fixtures/")
171
+ exporter.export_models([post], :user)
172
+ exporter.export_models([post.user])
173
+
174
+ # In our posts.yml file
175
+ posts_1234:
176
+ ...
177
+ user: users_5678
178
+
179
+ # In our users.yml file
180
+ users_5678:
181
+ ...
182
+
183
+ When we import these fixtures using the correct `insert_order` (`['users', 'posts']`), the newly imported post will belong to the newly imported user, no matter what their new ids are.
184
+
185
+ Currently, this works for belongs_to and has_and_belongs_to_many associations.
186
+
187
+ For has_and_belongs_to_many relation, add a field to one of the records, and the import will populate the join table.
188
+
189
+ The formatting of the fixture is quite flexible, and the value for this field can be either a list or comma-separated string containing either references or IDs of the associated records. In all cases, though, the value should match the association name. Note that in all cases below the key is `channels` and not `channel_ids`:
190
+
191
+ # In our users.yml file
192
+ users_5678:
193
+ channels: "1,2,3"
194
+
195
+ users_1234:
196
+ channels:
197
+ - channel_1
198
+ - bobs_cool_channel
199
+
200
+ # In our channels.yml file
201
+ channel_1:
202
+ ...
203
+
204
+ bobs_cool_channel:
205
+ ...
206
+
207
+ Also note it's not necessary to format your references the way the exporter does - `bobs_cool_channel` is a totally valid reference.
208
+
209
+ ### Templates
210
+
211
+ Templates allow you to export fixtures containing erb, that will be evaluated at the time of fixture import.
212
+
213
+ posts = Post.where(user_id: 5678
214
+ exporter = YourExporter.new("fixtures/")
215
+ exporter.export_models([post]) do |post|
216
+ { unique_promo_code: Template.new("<%= PromoCodeGenerator.unique_promo_code(#{post.id}) %>")
217
+ end
218
+
219
+ # In our fixtures/posts.yml file
220
+ posts_1234:
221
+ ...
222
+ user_id: 5678
223
+ unique_promo_code: <%= PromoCodeGenerator.unique_promo_code(1234) %>
224
+
225
+ In the example above, we'd be able to generate a new unique promo code for each post as we import them.
226
+
227
+
228
+ ## Motivation
229
+
230
+ One particular challenge when working with fixtures is describing associations between records. When they're in the database, records have unique primary keys, and associations are captured using foreign keys (references to the associated record's primary key). For example, a row in the `posts` table may have a column `user_id`. If it's value is `1234`, it indicated that `Post` belongs to the `User` with id `1234`.
231
+
232
+ How can we model associations when fixtures are removed from the database? It's not enough to just serialize the foreign keys, as we expect each record to have a different primary key each time it is imported into a database.
233
+
234
+ ActiveRecord::Fixtures answers this question in a way that is very effective for a test database, but that is not safe for a live database.
235
+
236
+ ### Here's how ActiveRecord::Fixtures work
237
+
238
+ Each record is assigned a label in its yml file. Primary key values are
239
+ assigned using a guid algorithm that maps a label to a consistent integer
240
+ between 1 and 2^30-1. Primary keys can then be assigned before saving any
241
+ records to the db.
242
+
243
+ Why would they do this? Because, this enables us to use labels in the
244
+ Fixture yml files to refer to associations. For example:
245
+
246
+ <users.yml>
247
+ bob:
248
+ username: thebob
249
+
250
+ <posts.yml>
251
+ hello:
252
+ message: Hello everyone!
253
+ user: bob
254
+
255
+
256
+
257
+ The ActiveRecord::Fixture system first converts every instance of `bob` and
258
+ `hello` into an integer using ActiveRecord::Fixture#identify, and then can
259
+ save the records IN ANY ORDER and know that all foreign keys will be valid.
260
+
261
+ There is a big problem with this. In a test db, each table is empty and so the
262
+ odds of inserting a few dozen records causing a primary key collision is
263
+ very small. However, for a production table with a hundred million rows, this
264
+ is no longer the case! Collisions abound and db insertion fails.
265
+
266
+ Also, auto-increment primary keys will continue from the LARGEST existing
267
+ primary key value. If we insert a record at 1,000,000,000 - we've reduced the
268
+ total number of records we can store in that table in half. Fine for a test db
269
+ but not ideal for production.
270
+
271
+
272
+ ### LiveFixtures work differently
273
+
274
+ Since we want to be able to take advantage of normal auto-increment behavior,
275
+ we cannot know the primary keys of each record before saving it to the db.
276
+ Instead, we save each record, and then maintain a mapping (`@label_to_id`)
277
+ from that record's label (`bob`), to its primary key (`213`). Later, when
278
+ another record (`hello`) references `bob`, we can use this mapping to look up
279
+ the primary key for `bob` before saving `hello`.
280
+
281
+ This means that the order we insert records into the db matters: `bob` must
282
+ be inserted before `hello`! This order is defined in INSERT_ORDER, and
283
+ reflected in the order of the `@table_names` array.
284
+
26
285
 
27
286
  ## Development
28
287
 
@@ -30,12 +289,13 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
30
289
 
31
290
  To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
32
291
 
292
+ Please remember to update the docs on your PR if you change anything. You can see the YARD docs live while you change them by running `yard server --reload`.
293
+
33
294
  ## Contributing
34
295
 
35
- Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/live_fixtures. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
296
+ Bug reports and pull requests are welcome on GitHub at https://github.com/NoRedInk/live_fixtures. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
36
297
 
37
298
 
38
299
  ## License
39
300
 
40
301
  The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
41
-
@@ -1,9 +1,11 @@
1
1
  require "live_fixtures/version"
2
2
  require "live_fixtures/import"
3
3
  require "live_fixtures/import/fixtures"
4
+ require "live_fixtures/import/insertion_order_computer"
4
5
  require "live_fixtures/export"
5
6
  require "live_fixtures/export/fixture"
6
7
  require "ruby-progressbar"
8
+ require "yaml"
7
9
 
8
10
  module LiveFixtures
9
11
  module_function
@@ -8,29 +8,61 @@
8
8
  # times for the same db table - that will overwrite the file each time!
9
9
 
10
10
  module LiveFixtures::Export
11
+ # Templates allow you to export fixtures containing erb, that will be evaluated at the time of fixture import.
12
+ # You should initialize them with a String containing the erb to evaluate, like
13
+ # @example A template with export and import times.
14
+ # Template.new("<%= \"I was exported at #{Time.now} and imported at \" + Time.now.to_s %>")
11
15
  Template = Struct.new(:code)
16
+
17
+ # References represent associations between fixtures, in the same way that foreign_keys do for records.
18
+ # These will be initialized for you based on the contents of `with_references` passed to `export_fixtures`.
19
+ # They will be initialized with the name of the association (a Symbol) and the particular associated model.
12
20
  Reference = Struct.new(:name, :value)
13
21
 
14
22
  private
15
23
 
24
+ # Specify the directory into which to export the yml files containing your fixtures.
25
+ # The directory will be created if it does not yet exist.
26
+ # @param dir [String] a path to a directory into which the fixtures will be exported.
16
27
  def set_export_dir(dir)
17
28
  @dir = dir
18
29
  FileUtils.mkdir_p(@dir) unless File.directory?(@dir)
19
30
  end
20
31
 
32
+ # Specify the options to use when exporting your fixtures.
33
+ # @param [Hash] opts export configuration options
34
+ # @option opts [Boolean] show_progress whether or not to show the progress bar
35
+ def set_export_options(**opts)
36
+ defaults = { show_progress: true }
37
+ @export_options = defaults.merge(opts)
38
+ end
39
+
40
+ # The options to use when exporting your fixtures.
41
+ # @return [Hash] export configuration options
42
+ def export_options
43
+ @export_options ||= set_export_options
44
+ end
45
+
21
46
  ##
22
47
  # Export models to a yml file named after the corresponding table.
48
+ # @param models [Enumerable] an Enumerable containing ActiveRecord models.
49
+ # @param with_references [Array<Symbol>] the associations whose foreign_keys should be replaced with references.
23
50
  #
24
51
  # Takes an optional block that will be invoked for each model.
25
52
  # The block should return a hash of attributes to be merged and
26
53
  # saved with the model's attributes.
54
+ # @yield [model] an optional block that will be invoked for each model.
55
+ # @yieldparam model [ActiveRecord::Base] each successive model.
56
+ # @yieldreturn [Hash{String => Object}] a hash of attributes to be merged and saved with the model's attributes.
27
57
  def export_fixtures(models, with_references = [])
28
58
  return unless models.present?
29
59
 
30
60
  table_name = models.first.class.table_name
31
61
  File.open(File.join(@dir, table_name + '.yml'), 'w') do |file|
32
62
 
33
- ProgressBarIterator.new(models).each do |model|
63
+ iterator = export_options[:show_progress] ? ProgressBarIterator : SimpleIterator
64
+
65
+ iterator.new(models).each do |model|
34
66
  more_attributes = block_given? ? yield(model) : {}
35
67
  file.write Fixture.to_yaml(model, with_references, more_attributes)
36
68
  end
@@ -54,4 +86,17 @@ module LiveFixtures::Export
54
86
  end
55
87
  end
56
88
  end
89
+
90
+ class SimpleIterator
91
+ def initialize(models)
92
+ @models = models
93
+ end
94
+
95
+ def each
96
+ puts @models.first.class.name
97
+ @models.each do |model|
98
+ yield model
99
+ end
100
+ end
101
+ end
57
102
  end
@@ -1,5 +1,13 @@
1
+ # Exposes functionality to serialize an ActiveRecord record (a model) into a
2
+ # YAML fixture.
1
3
  module LiveFixtures::Export::Fixture
2
4
  module_function
5
+ # YAML-Serializes the provided model, including any references and additional
6
+ # attribtues.
7
+ # @param model [ActiveRecord::Base] an ActiveRecord record to serialize
8
+ # @param references [Symbol, Array<Symbol>] the names of associations whose foreign_keys should be replaced with references
9
+ # @param more_attributes [Hash{String => Time, DateTime, Date, Hash, String, LiveFixtures::Export::Template, LiveFixtures::Export::Reference, #to_s}] a hash of additional attributes to serialize with each record.
10
+ # @return [String] the model serialized in YAML, with specified foreign_keys replaced by references, including additional attributes.
3
11
  def to_yaml(model, references = [], more_attributes = {})
4
12
  table_name = model.class.table_name
5
13
 
@@ -28,29 +36,44 @@ module LiveFixtures::Export::Fixture
28
36
  end
29
37
 
30
38
  private_class_method def yml_attributes(model, more_attributes)
31
- model.attributes.merge(more_attributes).map do |name, value|
32
- next if %w{id}.include? name
39
+ model.attributes.except("id").merge(more_attributes).map do |name, value|
33
40
  next if value.nil?
34
41
 
35
- yml_value ||= case value
36
- when Time, DateTime
37
- value.utc.to_s(:db)
38
- when Date
39
- value.to_s(:db)
40
- when Hash
41
- value.to_yaml.inspect
42
- when String
43
- value.inspect
44
- when LiveFixtures::Export::Template
45
- value.code
46
- when LiveFixtures::Export::Reference
47
- name, value = value.name, value.value
48
- "#{value.class.table_name}_#{value.id}"
49
- else
50
- value.to_s
51
- end
52
-
53
- "#{name}: " + yml_value
42
+ serialize_attribute(model, name, value)
54
43
  end.compact.join("\n ")
55
44
  end
45
+
46
+ private_class_method def serialize_attribute(model, name, value)
47
+ attribute_type = model.class.type_for_attribute(name)
48
+
49
+ if attribute_type.is_a?(ActiveRecord::Type::Serialized)
50
+ value = attribute_type.serialize(value) unless value.is_a?(String)
51
+
52
+ "#{name}: |-\n#{value.to_s.indent(4)}" unless value.nil?
53
+ elsif value.is_a? LiveFixtures::Export::Reference
54
+ "#{value.name}: #{yml_value(value)}"
55
+ else
56
+ "#{name}: #{yml_value(value)}"
57
+ end
58
+ end
59
+
60
+ private_class_method def yml_value(value)
61
+ case value
62
+ when Time, DateTime
63
+ value.utc.to_s(:db)
64
+ when Date
65
+ value.to_s(:db)
66
+ when Hash
67
+ value.to_yaml.inspect
68
+ when String
69
+ value.inspect
70
+ when LiveFixtures::Export::Template
71
+ value.code
72
+ when LiveFixtures::Export::Reference
73
+ reference_value = value.value
74
+ "#{reference_value.class.table_name}_#{reference_value.id}"
75
+ else
76
+ value.to_s
77
+ end
78
+ end
56
79
  end
@@ -1,82 +1,58 @@
1
- # ActiveRecord::Fixtures are a powerful way of populating data in a db;
2
- # however, its strategy for handling primary keys and associations is
3
- # UNACCEPTABLE for use with a production db. LiveFixtures works around this.
4
- #
5
- #
6
- ########### Here's how ActiveRecord::Fixtures work ###########################
7
- #
8
- # Each record is assigned a label in its yml file. Primary key values are
9
- # assigned using a guid algorithm that maps a label to a consistent integer
10
- # between 1 and 2^30-1. Primary keys can then be assigned before saving any
11
- # records to the db.
12
- #
13
- # Why would they do this? Because, this enables us to use labels in the
14
- # Fixture yml files to refer to associations. For example:
15
- #
16
- # <users.yml>
17
- # bob:
18
- # username: thebob
19
- #
20
- # <posts.yml>
21
- # hello:
22
- # message: Hello everyone!
23
- # user: bob
24
- #
25
- # The ActiveRecord::Fixture system first converts every instance of `bob` and
26
- # `hello` into an integer using ActiveRecord::Fixture#identify, and then can
27
- # save the records IN ANY ORDER and know that all foreign keys will be valid.
28
- #
29
- # There is a big problem with this. In a test db, each table is empty and so the
30
- # odds of inserting a few dozen records causing a primary key collision is
31
- # very small. However, for a production table with a hundred million rows, this
32
- # is no longer the case! Collisions abound and db insertion fails.
33
- #
34
- # Also, autoincrement primary keys will continue from the LARGEST existing
35
- # primary key value. If we insert a record at 1,000,000,000 - we've reduced the
36
- # total number of records we can store in that table in half. Fine for a test db
37
- # but not ideal for production.
38
- #
39
- #
40
- ########### LiveFixtures work differently ####################################
41
- #
42
- # Since we want to be able to take advantage of normal autoincrement behavior,
43
- # we cannot know the primary keys of each record before saving it to the db.
44
- # Instead, we save each record, and then maintain a mapping (`@label_to_id`)
45
- # from that record's label (`bob`), to its primary key (`213`). Later, when
46
- # another record (`hello`) references `bob`, we can use this mapping to look up
47
- # the primary key for `bob` before saving `hello`.
48
- #
49
- # This means that the order we insert records into the db matters: `bob` must
50
- # be inserted before `hello`! This order is defined in INSERT_ORDER, and
51
- # reflected in the order of the `@table_names` array.
52
-
1
+ # An object that facilitates the import of fixtures into a database.
53
2
  class LiveFixtures::Import
54
3
  NO_LABEL = nil
55
4
 
56
- def initialize(root_path, insert_order)
5
+ # Returns the insert order that was specified in the constructor or
6
+ # the inferred one if none was specified.
7
+ attr_reader :insert_order
8
+
9
+ # Instantiate a new Import with the directory containing your fixtures, and
10
+ # the order in which to import them. The order should ensure fixtures
11
+ # containing references to another fixture are imported AFTER the referenced
12
+ # fixture.
13
+ # @raise [ArgumentError] raises an argument error if not every element in the insert_order has a corresponding yml file.
14
+ # @param root_path [String] path to the directory containing the yml files to import.
15
+ # @param insert_order [Array<String> | Nil] a list of yml files (without .yml extension) in the order they should be imported, or `nil` if these order is to be inferred by this class.
16
+ # @param class_names [Hash{Symbol => String}] a mapping table name => Model class, for any that don't follow convention.
17
+ # @param [Hash] opts export configuration options
18
+ # @option opts [Boolean] show_progress whether or not to show the progress bar
19
+ # @option opts [Boolean] skip_missing_tables when false, an error will be raised if a yaml file isn't found for each table in insert_order
20
+ # @option opts [Boolean] skip_missing_refs when false, an error will be raised if an ID isn't found for a label.
21
+ # @return [LiveFixtures::Import] an importer
22
+ # @see LiveFixtures::Export::Reference
23
+ def initialize(root_path, insert_order = nil, class_names = {}, **opts)
24
+ defaut_options = { show_progress: true, skip_missing_tables: false, skip_missing_refs: false }
25
+ @options = defaut_options.merge(opts)
57
26
  @root_path = root_path
58
27
  @table_names = Dir.glob(File.join(@root_path, '{*,**}/*.yml')).map do |filepath|
59
28
  File.basename filepath, ".yml"
60
29
  end
61
- @table_names = insert_order.select {|table_name| @table_names.include? table_name}
62
- if @table_names.size < insert_order.size
63
- raise ArgumentError, "table(s) mentioned in `insert_order` which has no yml file to import: #{insert_order - @table_names}"
30
+
31
+ @class_names = class_names
32
+ @table_names.each { |n|
33
+ @class_names[n.tr('/', '_').to_sym] ||= n.classify if n.include?('/')
34
+ }
35
+
36
+ @insert_order = insert_order
37
+ @insert_order ||= InsertionOrderComputer.compute(@table_names, @class_names, compute_polymorphic_associations)
38
+
39
+ @table_names = @insert_order.select {|table_name| @table_names.include? table_name}
40
+ if @table_names.size < @insert_order.size && !@options[:skip_missing_tables]
41
+ raise ArgumentError, "table(s) mentioned in `insert_order` which has no yml file to import: #{@insert_order - @table_names}"
64
42
  end
65
43
  @label_to_id = {}
66
44
  end
67
45
 
68
- # https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L496
46
+ # Within a transaction, import all the fixtures into the database.
47
+ #
69
48
  # The very similar method: ActiveRecord::FixtureSet.create_fixtures has the
70
49
  # unfortunate side effect of truncating each table!!
71
50
  #
72
51
  # Therefore, we have reproduced the relevant sections here, without DELETEs,
73
- # with calling `LF::Import::Fixtures#each_table_row_with_label` instead of
52
+ # with calling {LiveFixtures::Import::Fixtures#each_table_row_with_label} instead of
74
53
  # `AR::Fixtures#table_rows`, and using those labels to populate `@label_to_id`.
75
- def import_all(class_names = {})
76
- @table_names.each { |n|
77
- class_names[n.tr('/', '_').to_sym] ||= n.classify if n.include?('/')
78
- }
79
-
54
+ # @see https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L496
55
+ def import_all
80
56
  connection = ActiveRecord::Base.connection
81
57
 
82
58
  files_to_read = @table_names
@@ -85,22 +61,70 @@ class LiveFixtures::Import
85
61
  connection.transaction(requires_new: true) do
86
62
  files_to_read.each do |path|
87
63
  table_name = path.tr '/', '_'
88
- class_name = class_names[table_name.to_sym] || table_name.classify
64
+ class_name = @class_names[table_name.to_sym] || table_name.classify
89
65
 
90
66
  ff = Fixtures.new(connection,
91
67
  table_name,
92
68
  class_name,
93
69
  ::File.join(@root_path, path),
94
- @label_to_id)
70
+ @label_to_id,
71
+ skip_missing_refs: @options[:skip_missing_refs])
95
72
 
96
73
  conn = ff.model_connection || connection
97
- ProgressBarIterator.new(ff).each do |table_name, label, row|
74
+ iterator = @options[:show_progress] ? ProgressBarIterator : SimpleIterator
75
+ iterator.new(ff).each do |table_name, label, row|
98
76
  conn.insert_fixture(row, table_name)
99
- @label_to_id[label] = conn.last_inserted_id(table_name) unless label == NO_LABEL
77
+ @label_to_id[label] = conn.send(:last_inserted_id, table_name) unless label == NO_LABEL
78
+ end
79
+ end
80
+ end
81
+ end
82
+ end
83
+
84
+ private
85
+
86
+ # Here we go through each of the fixture YAML files to see what polymorphic
87
+ # dependencies exist for each of the models.
88
+ # We do this by inspecting the value of any field that ends with `_type`,
89
+ # for example `author_type`, `assignment_type`, etc.
90
+ # Becuase we can't know all the possible types of a polymorphic association
91
+ # we compute them from the YAML file contents.
92
+ # Returns a Hash[Class => Set[Class]]
93
+ def compute_polymorphic_associations
94
+ polymorphic_associations = Hash.new { |h, k| h[k] = Set.new }
95
+
96
+ connection = ActiveRecord::Base.connection
97
+ files_to_read = @table_names
98
+
99
+ files_to_read.each do |path|
100
+ table_name = path.tr '/', '_'
101
+ class_name = @class_names[table_name.to_sym] || table_name.classify
102
+
103
+ # Here we use the yaml file and YAML.load instead of ActiveRecord::FixtureSet.new
104
+ # because it's faster and we can also check whether we actually need to
105
+ # load the file: only if it includes "_type" in it, otherwise there will be
106
+ # no polymorphic types in there.
107
+
108
+ filename = ::File.join(@root_path, "#{path}.yml")
109
+ file = File.read(filename)
110
+ next unless file =~ /_type/
111
+
112
+ yaml = YAML.load(file)
113
+ yaml.each do |key, object|
114
+ object.each do |field, value|
115
+ next unless field.ends_with?("_type")
116
+
117
+ begin
118
+ polymorphic_associations[class_name.constantize] << value.constantize
119
+ rescue NameError
120
+ # It might be the case that the `..._type` field doesn't actually
121
+ # refer to a type name, so we just ignore it.
100
122
  end
101
123
  end
102
124
  end
103
125
  end
126
+
127
+ polymorphic_associations
104
128
  end
105
129
 
106
130
  class ProgressBarIterator
@@ -120,4 +144,17 @@ class LiveFixtures::Import
120
144
  @bar.finish
121
145
  end
122
146
  end
147
+
148
+ class SimpleIterator
149
+ def initialize(ff)
150
+ @ff = ff
151
+ end
152
+
153
+ def each
154
+ puts @ff.model_class.name
155
+ @ff.each_table_row_with_label do |*args|
156
+ yield(*args)
157
+ end
158
+ end
159
+ end
123
160
  end
@@ -1,13 +1,23 @@
1
1
  require 'active_record/fixtures'
2
2
 
3
- #rubocop:disable Style/PerlBackrefs
4
-
5
3
  class LiveFixtures::Import
4
+ # A labeled reference was not found.
5
+ # Maybe the referenced model was not exported, or the insert order attempted
6
+ # to import the reference before the referenced model?
7
+ LiveFixtures::MissingReferenceError = Class.new(KeyError)
8
+
6
9
  class Fixtures
7
10
  delegate :model_class, :table_name, :fixtures, to: :ar_fixtures
11
+ # ActiveRecord::FixtureSet for delegation
8
12
  attr_reader :ar_fixtures
9
13
 
10
- def initialize(connection, table_name, class_name, filepath, label_to_id)
14
+ # @param connection [ActiveRecord::ConnectionAdapters::AbstractAdapter] connection to the database into which to import the data.
15
+ # @param table_name [String] name of the database table to populate with models
16
+ # @param class_name [Constant] the model's class name
17
+ # @param filepath [String] path to the yml file containing the fixtures
18
+ # @param label_to_id [Hash{String => Int}] map from a reference's label to its new id.
19
+ def initialize(connection, table_name, class_name, filepath, label_to_id, skip_missing_refs: false)
20
+ @skip_missing_refs = skip_missing_refs
11
21
  @ar_fixtures = ActiveRecord::FixtureSet.new connection,
12
22
  table_name,
13
23
  class_name,
@@ -15,12 +25,16 @@ class LiveFixtures::Import
15
25
  @label_to_id = label_to_id
16
26
  end
17
27
 
18
- # https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L611
19
28
  # Rewritten to take advantage of @label_to_id instead of AR::FixtureSet#identify,
20
29
  # and to make an iterator.
21
30
  #
31
+ # @yieldparam table_name [String] the database table's name
32
+ # @yieldparam label [String] the label for the model being currently imported
33
+ # @yieldparam row [Hash{String => Value}] the model's attributes to be imported
22
34
  # Iterator which yields [table_name, label, row] for each fixture
23
- # (and for any implicit join table records)
35
+ # (and for any implicit join table records). The block is expected to insert
36
+ # the row and update @label_to_id with the record's newly assigned id.
37
+ # @see https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L611
24
38
  def each_table_row_with_label
25
39
  join_table_rows = Hash.new { |h,table| h[table] = [] }
26
40
 
@@ -55,7 +69,7 @@ class LiveFixtures::Import
55
69
  rows.each do |targets:, association:, label:|
56
70
  targets.each do |target|
57
71
  assoc_fk = @label_to_id[target] || target
58
- row = { association.foreign_key => @label_to_id[label],
72
+ row = { association.foreign_key => fetch_id_for_label(label),
59
73
  association.association_foreign_key => assoc_fk }
60
74
  yield [table_name, NO_LABEL, row]
61
75
  end
@@ -95,7 +109,25 @@ class LiveFixtures::Import
95
109
  row[association.foreign_type] = $1
96
110
  end
97
111
 
98
- row[fk_name] = @label_to_id[value]
112
+ row[fk_name] = fetch_id_for_label(value)
113
+ end
114
+
115
+ # Uses the underlying map of labels to return the referenced model's newly
116
+ # assigned ID.
117
+ # @raise [LiveFixtures::MissingReferenceError] if the label isn't found.
118
+ # @param label_to_fetch [String] the label of the referenced model.
119
+ # @return [Integer] the newly assigned ID of the referenced model.
120
+ def fetch_id_for_label(label_to_fetch)
121
+ @label_to_id.fetch(label_to_fetch)
122
+ rescue KeyError
123
+ return nil if @skip_missing_refs
124
+
125
+ raise LiveFixtures::MissingReferenceError, <<-ERROR.squish
126
+ Unable to find ID for model referenced by label #{label_to_fetch} while
127
+ importing #{model_class} from #{table_name}.yml. Perhaps it isn't included
128
+ in these fixtures or it is too late in the insert_order and has not yet
129
+ been imported.
130
+ ERROR
99
131
  end
100
132
 
101
133
  private :ar_fixtures
@@ -0,0 +1,127 @@
1
+ class LiveFixtures::Import
2
+ # :nodoc:
3
+ class InsertionOrderComputer
4
+ # :nodoc:
5
+ class Node
6
+ attr_reader :path
7
+ attr_reader :class_name
8
+ attr_reader :klass
9
+
10
+ # The classes this node depends on
11
+ attr_reader :dependencies
12
+
13
+ def initialize(path, class_name, klass)
14
+ @path = path
15
+ @class_name = class_name
16
+ @klass = klass
17
+ @dependencies = Set.new
18
+ end
19
+ end
20
+
21
+ def self.compute(table_names, class_names = {}, polymorphic_associations = {})
22
+ new(table_names, class_names, polymorphic_associations).compute
23
+ end
24
+
25
+ def initialize(table_names, class_names = {}, polymorphic_associations = {})
26
+ @table_names = table_names
27
+ @class_names = class_names
28
+ @polymorphic_associations = polymorphic_associations
29
+ end
30
+
31
+ def compute
32
+ nodes = build_nodes
33
+ compute_insert_order(nodes)
34
+ end
35
+
36
+ private
37
+
38
+ # Builds an Array of Nodes, each containing dependencies to other nodes
39
+ # using their class names.
40
+ def build_nodes
41
+ # Create a Hash[Class => Node] for each table/class
42
+ nodes = {}
43
+ @table_names.each do |path|
44
+ table_name = path.tr "/", "_"
45
+ class_name = @class_names[table_name.to_sym] || table_name.classify
46
+ klass = class_name.constantize
47
+ nodes[klass] = Node.new(path, class_name, klass)
48
+ end
49
+
50
+ # First iniitalize dependencies from polymorphic associations that we
51
+ # explicitly found in the yaml files.
52
+ @polymorphic_associations.each do |klass, associations|
53
+ associations.each do |association|
54
+ node = nodes[klass]
55
+ next unless node
56
+ next unless nodes.key?(association)
57
+
58
+ node.dependencies << association
59
+ end
60
+ end
61
+
62
+ # Compute dependencies between nodes/classes by reflecting on their
63
+ # ActiveRecord associations.
64
+ nodes.each do |_, node|
65
+ klass = node.klass
66
+ klass.reflect_on_all_associations.each do |assoc|
67
+ # We can't handle polymorphic associations, but the concrete types
68
+ # should have been deduced from the yaml files contents
69
+ next if assoc.polymorphic?
70
+
71
+ # Don't add a dependency if the class is not in the given table names
72
+ next unless nodes.key?(assoc.klass)
73
+
74
+ # A class might depend on itself, but we don't add it as a dependency
75
+ # because otherwise we'll never make it (the class can probably be created
76
+ # just fine and these dependencies are optional/nilable)
77
+ next if klass == assoc.klass
78
+
79
+ case assoc.macro
80
+ when :belongs_to
81
+ node.dependencies << assoc.klass
82
+ when :has_one, :has_many
83
+ # Skip `through` association becuase it will be already computed
84
+ # for the related `has_one`/`has_many` association
85
+ next if assoc.options[:through]
86
+
87
+ nodes[assoc.klass].dependencies << klass
88
+ end
89
+ end
90
+ end
91
+
92
+ # Finally sort all values by name for consistent results
93
+ nodes.values.sort_by { |node| node.klass.name }
94
+ end
95
+
96
+ def compute_insert_order(nodes)
97
+ insert_order = []
98
+
99
+ until nodes.empty?
100
+ # Pick a node that has no dependencies
101
+ free_node = nodes.find { |node| node.dependencies.empty? }
102
+
103
+ if free_node.nil?
104
+ msg = "Can't compute an insert order.\n\n"
105
+ msg << "These models seem to depend on each other:\n"
106
+ nodes.each do |node|
107
+ msg << " #{node.klass.name}\n"
108
+ msg << " - depends on: #{node.dependencies.map(&:name).join(", ")}\n"
109
+ end
110
+ raise msg
111
+ end
112
+
113
+ insert_order << free_node.path
114
+
115
+ # Delete this node from the other nodes' dependencies
116
+ nodes.each do |node|
117
+ node.dependencies.delete(free_node.klass)
118
+ end
119
+
120
+ # And delete this node because we are done with it
121
+ nodes.delete(free_node)
122
+ end
123
+
124
+ insert_order
125
+ end
126
+ end
127
+ end
@@ -1,3 +1,3 @@
1
1
  module LiveFixtures
2
- VERSION = "0.2.0"
2
+ VERSION = "2.0.0"
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: live_fixtures
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 2.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - jleven
8
- autorequire:
8
+ autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2017-05-09 00:00:00.000000000 Z
11
+ date: 2020-11-11 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activerecord
@@ -16,14 +16,14 @@ dependencies:
16
16
  requirements:
17
17
  - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: '4.2'
19
+ version: '5.2'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: '4.2'
26
+ version: '5.2'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: ruby-progressbar
29
29
  requirement: !ruby/object:Gem::Requirement
@@ -44,56 +44,56 @@ dependencies:
44
44
  requirements:
45
45
  - - "~>"
46
46
  - !ruby/object:Gem::Version
47
- version: '1.11'
47
+ version: '2.0'
48
48
  type: :development
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
- version: '1.11'
54
+ version: '2.0'
55
55
  - !ruby/object:Gem::Dependency
56
56
  name: rake
57
57
  requirement: !ruby/object:Gem::Requirement
58
58
  requirements:
59
59
  - - "~>"
60
60
  - !ruby/object:Gem::Version
61
- version: '10.0'
61
+ version: '12.3'
62
62
  type: :development
63
63
  prerelease: false
64
64
  version_requirements: !ruby/object:Gem::Requirement
65
65
  requirements:
66
66
  - - "~>"
67
67
  - !ruby/object:Gem::Version
68
- version: '10.0'
68
+ version: '12.3'
69
69
  - !ruby/object:Gem::Dependency
70
70
  name: rspec
71
71
  requirement: !ruby/object:Gem::Requirement
72
72
  requirements:
73
73
  - - "~>"
74
74
  - !ruby/object:Gem::Version
75
- version: '3.0'
75
+ version: '3.8'
76
76
  type: :development
77
77
  prerelease: false
78
78
  version_requirements: !ruby/object:Gem::Requirement
79
79
  requirements:
80
80
  - - "~>"
81
81
  - !ruby/object:Gem::Version
82
- version: '3.0'
82
+ version: '3.8'
83
83
  - !ruby/object:Gem::Dependency
84
84
  name: temping
85
85
  requirement: !ruby/object:Gem::Requirement
86
86
  requirements:
87
87
  - - "~>"
88
88
  - !ruby/object:Gem::Version
89
- version: '3.0'
89
+ version: '3.10'
90
90
  type: :development
91
91
  prerelease: false
92
92
  version_requirements: !ruby/object:Gem::Requirement
93
93
  requirements:
94
94
  - - "~>"
95
95
  - !ruby/object:Gem::Version
96
- version: '3.0'
96
+ version: '3.10'
97
97
  - !ruby/object:Gem::Dependency
98
98
  name: byebug
99
99
  requirement: !ruby/object:Gem::Requirement
@@ -110,6 +110,62 @@ dependencies:
110
110
  version: '0'
111
111
  - !ruby/object:Gem::Dependency
112
112
  name: sqlite3
113
+ requirement: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - "~>"
116
+ - !ruby/object:Gem::Version
117
+ version: 1.3.13
118
+ type: :development
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - "~>"
123
+ - !ruby/object:Gem::Version
124
+ version: 1.3.13
125
+ - !ruby/object:Gem::Dependency
126
+ name: mysql2
127
+ requirement: !ruby/object:Gem::Requirement
128
+ requirements:
129
+ - - ">="
130
+ - !ruby/object:Gem::Version
131
+ version: '0'
132
+ type: :development
133
+ prerelease: false
134
+ version_requirements: !ruby/object:Gem::Requirement
135
+ requirements:
136
+ - - ">="
137
+ - !ruby/object:Gem::Version
138
+ version: '0'
139
+ - !ruby/object:Gem::Dependency
140
+ name: pg
141
+ requirement: !ruby/object:Gem::Requirement
142
+ requirements:
143
+ - - ">="
144
+ - !ruby/object:Gem::Version
145
+ version: '0'
146
+ type: :development
147
+ prerelease: false
148
+ version_requirements: !ruby/object:Gem::Requirement
149
+ requirements:
150
+ - - ">="
151
+ - !ruby/object:Gem::Version
152
+ version: '0'
153
+ - !ruby/object:Gem::Dependency
154
+ name: yard
155
+ requirement: !ruby/object:Gem::Requirement
156
+ requirements:
157
+ - - ">="
158
+ - !ruby/object:Gem::Version
159
+ version: '0'
160
+ type: :development
161
+ prerelease: false
162
+ version_requirements: !ruby/object:Gem::Requirement
163
+ requirements:
164
+ - - ">="
165
+ - !ruby/object:Gem::Version
166
+ version: '0'
167
+ - !ruby/object:Gem::Dependency
168
+ name: reek
113
169
  requirement: !ruby/object:Gem::Requirement
114
170
  requirements:
115
171
  - - ">="
@@ -122,7 +178,7 @@ dependencies:
122
178
  - - ">="
123
179
  - !ruby/object:Gem::Version
124
180
  version: '0'
125
- description:
181
+ description:
126
182
  email:
127
183
  - josh@noredink.com
128
184
  executables: []
@@ -137,12 +193,13 @@ files:
137
193
  - lib/live_fixtures/export/fixture.rb
138
194
  - lib/live_fixtures/import.rb
139
195
  - lib/live_fixtures/import/fixtures.rb
196
+ - lib/live_fixtures/import/insertion_order_computer.rb
140
197
  - lib/live_fixtures/version.rb
141
- homepage:
198
+ homepage:
142
199
  licenses:
143
200
  - MIT
144
201
  metadata: {}
145
- post_install_message:
202
+ post_install_message:
146
203
  rdoc_options: []
147
204
  require_paths:
148
205
  - lib
@@ -157,9 +214,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
157
214
  - !ruby/object:Gem::Version
158
215
  version: '0'
159
216
  requirements: []
160
- rubyforge_project:
161
- rubygems_version: 2.6.12
162
- signing_key:
217
+ rubygems_version: 3.0.3
218
+ signing_key:
163
219
  specification_version: 4
164
220
  summary: Tools for exporting and importing between databases managed by ActiveRecord.
165
221
  test_files: []