populator 0.2.5 → 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -1,8 +1,15 @@
1
- 0.2.5 (October 1st, 2008)
1
+ 1.0.0 (January 25, 2011)
2
2
 
3
- * compatibility with RubyGems 1.3
3
+ * Fixing SQLite adapter to work in Rails 3
4
+
5
+ * Improving PostgreSQL adapter (thanks phlawski and zehkae)
6
+
7
+ * adding Oracle adapter (thanks Andrew N)
8
+
9
+ * Upgrading to RSpec 2 and Rails 3 in specs
10
+
11
+ * New Gemfile and .rvmrc file to make development and running specs easier
4
12
 
5
- * mass assign attributes with "attributes=" method
6
13
 
7
14
  0.2.4 (September 9th, 2008)
8
15
 
@@ -10,24 +17,28 @@
10
17
 
11
18
  * adding Populator.paragraphs to generate paragraphs of text
12
19
 
20
+
13
21
  0.2.3 (September 2nd, 2008)
14
22
 
15
23
  * support single table inhertance by setting inheritance_column to class name
16
24
 
17
25
  * support custom primary_key in model if they don't use "id"
18
26
 
27
+
19
28
  0.2.2 (September 1st, 2008)
20
29
 
21
30
  * performance improvements
22
31
 
23
32
  * improving inline documentation
24
33
 
34
+
25
35
  0.2.1 (August 30th, 2008)
26
36
 
27
37
  * wrap sqlite inserts in transaction to improve performance
28
38
 
29
39
  * default created_at/on and updated_at/on columns to current time
30
40
 
41
+
31
42
  0.2.0 (August 30th, 2008)
32
43
 
33
44
  * adding :per_query option to limit how many inserts are made per query
@@ -38,6 +49,7 @@
38
49
 
39
50
  * adding Populator.words to fetch some random words
40
51
 
52
+
41
53
  0.1.0 (August 27th, 2008)
42
54
 
43
55
  * initial release
data/Gemfile ADDED
@@ -0,0 +1,9 @@
1
+ source "http://rubygems.org"
2
+
3
+ # TODO change depending on adapter (?)
4
+ gem "sqlite3-ruby", :require => "sqlite3"
5
+ gem "mysql2"
6
+ gem "pg"
7
+ gem "activerecord", :require => "active_record"
8
+
9
+ gemspec
@@ -0,0 +1,99 @@
1
+ PATH
2
+ remote: .
3
+ specs:
4
+ populator (1.0.0.beta1)
5
+
6
+ GEM
7
+ remote: http://rubygems.org/
8
+ specs:
9
+ abstract (1.0.0)
10
+ actionmailer (3.0.3)
11
+ actionpack (= 3.0.3)
12
+ mail (~> 2.2.9)
13
+ actionpack (3.0.3)
14
+ activemodel (= 3.0.3)
15
+ activesupport (= 3.0.3)
16
+ builder (~> 2.1.2)
17
+ erubis (~> 2.6.6)
18
+ i18n (~> 0.4)
19
+ rack (~> 1.2.1)
20
+ rack-mount (~> 0.6.13)
21
+ rack-test (~> 0.5.6)
22
+ tzinfo (~> 0.3.23)
23
+ activemodel (3.0.3)
24
+ activesupport (= 3.0.3)
25
+ builder (~> 2.1.2)
26
+ i18n (~> 0.4)
27
+ activerecord (3.0.3)
28
+ activemodel (= 3.0.3)
29
+ activesupport (= 3.0.3)
30
+ arel (~> 2.0.2)
31
+ tzinfo (~> 0.3.23)
32
+ activeresource (3.0.3)
33
+ activemodel (= 3.0.3)
34
+ activesupport (= 3.0.3)
35
+ activesupport (3.0.3)
36
+ arel (2.0.7)
37
+ builder (2.1.2)
38
+ diff-lcs (1.1.2)
39
+ erubis (2.6.6)
40
+ abstract (>= 1.0.0)
41
+ i18n (0.5.0)
42
+ mail (2.2.14)
43
+ activesupport (>= 2.3.6)
44
+ i18n (>= 0.4.0)
45
+ mime-types (~> 1.16)
46
+ treetop (~> 1.4.8)
47
+ mime-types (1.16)
48
+ mocha (0.9.10)
49
+ rake
50
+ mysql2 (0.2.6)
51
+ pg (0.10.1)
52
+ polyglot (0.3.1)
53
+ rack (1.2.1)
54
+ rack-mount (0.6.13)
55
+ rack (>= 1.0.0)
56
+ rack-test (0.5.7)
57
+ rack (>= 1.0)
58
+ rails (3.0.3)
59
+ actionmailer (= 3.0.3)
60
+ actionpack (= 3.0.3)
61
+ activerecord (= 3.0.3)
62
+ activeresource (= 3.0.3)
63
+ activesupport (= 3.0.3)
64
+ bundler (~> 1.0)
65
+ railties (= 3.0.3)
66
+ railties (3.0.3)
67
+ actionpack (= 3.0.3)
68
+ activesupport (= 3.0.3)
69
+ rake (>= 0.8.7)
70
+ thor (~> 0.14.4)
71
+ rake (0.8.7)
72
+ rspec (2.1.0)
73
+ rspec-core (~> 2.1.0)
74
+ rspec-expectations (~> 2.1.0)
75
+ rspec-mocks (~> 2.1.0)
76
+ rspec-core (2.1.0)
77
+ rspec-expectations (2.1.0)
78
+ diff-lcs (~> 1.1.2)
79
+ rspec-mocks (2.1.0)
80
+ sqlite3 (1.3.3)
81
+ sqlite3-ruby (1.3.3)
82
+ sqlite3 (>= 1.3.3)
83
+ thor (0.14.6)
84
+ treetop (1.4.9)
85
+ polyglot (>= 0.3.1)
86
+ tzinfo (0.3.24)
87
+
88
+ PLATFORMS
89
+ ruby
90
+
91
+ DEPENDENCIES
92
+ activerecord
93
+ mocha (~> 0.9.10)
94
+ mysql2
95
+ pg
96
+ populator!
97
+ rails (~> 3.0.3)
98
+ rspec (~> 2.1.0)
99
+ sqlite3-ruby
data/LICENSE CHANGED
@@ -1,4 +1,4 @@
1
- Copyright (c) 2008 Ryan Bates
1
+ Copyright (c) 2011 Ryan Bates
2
2
 
3
3
  Permission is hereby granted, free of charge, to any person obtaining
4
4
  a copy of this software and associated documentation files (the
@@ -1,37 +1,33 @@
1
1
  = Populator
2
2
 
3
- Populate an Active Record database with mass insert.
3
+ RDocs[http://rdoc.info/projects/ryanb/populator] | Screencast[http://railscasts.com/episodes/126-populating-a-database]
4
4
 
5
- You can find the rdocs at http://populator.rubyforge.org.
5
+ Populate an Active Record database with mass insertion.
6
6
 
7
- Special thanks to Zach Dennis for his ar-extensions gem which some of
8
- this code is loosely based on.
9
7
 
8
+ == Installation
10
9
 
11
- == Install
10
+ <b>Rails 3</b> support is currently being worked on. Stay tuned.
12
11
 
13
- Install the gem:
12
+ In <b>Rails 2</b>, install the gem.
14
13
 
15
14
  gem install populator
16
15
 
17
- And then load it in your project:
16
+ And then load it in a rake task or elsewhere.
18
17
 
19
- require 'populator'
18
+ require "populator"
20
19
 
21
20
 
22
21
  == Usage
23
22
 
24
- This gem adds a "populate" method to all Active Record models. Pass the
25
- number of records you want to create along with a block. In the block
26
- you can set the column values for each record.
23
+ This gem adds a "populate" method to all Active Record models. Pass the number of records you want to create along with a block. In the block you can set the column values for each record.
27
24
 
28
25
  Person.populate(3000) do |person|
29
26
  person.first_name = "John"
30
27
  person.last_name = "Smith"
31
28
  end
32
29
 
33
- This will do a mass insert into the database so it is very fast.
34
- The person object contains the "id" so you can set up associations.
30
+ This will do a mass insert into the database so it is very fast. The person object contains the "id" so you can set up associations.
35
31
 
36
32
  Person.populate(3000) do |person|
37
33
  person.first_name = "John"
@@ -50,11 +46,9 @@ Passing a range or array of values will randomly select one.
50
46
  person.annual_income = 10000..200000
51
47
  end
52
48
 
53
- This will create 1000 to 5000 men or women with the annual income
54
- between 10,000 and 200,000.
49
+ This will create 1000 to 5000 men or women with the annual income between 10,000 and 200,000.
55
50
 
56
- You can pass a :per_query option to limit how many records are saved
57
- per query. This defaults to 1000.
51
+ You can pass a :per_query option to limit how many records are saved per query. This defaults to 1000.
58
52
 
59
53
  Person.populate(2000, :per_query => 100)
60
54
 
@@ -65,27 +59,21 @@ If you need to generate fake data, there are a few methods to do this.
65
59
  Populator.sentences(5) # generates 5 sentences
66
60
  Populator.paragraphs(3) # generates 3 paragraphs
67
61
 
68
- For fancier data generation, try the Faker gem.
69
-
70
- http://faker.rubyforge.org
62
+ For fancier data generation, try the {Faker gem}[http://faker.rubyforge.org].
71
63
 
72
64
 
73
65
  == Important
74
66
 
75
- For performance reasons, this gem does not use actual instances of the
76
- model. This means validations and callbacks are bypassed. It is up to
77
- you to ensure you're adding valid data.
67
+ For performance reasons, this gem does not use actual instances of the model. This means validations and callbacks are bypassed. It is up to you to ensure you're adding valid data.
78
68
 
79
69
 
80
70
  == Development
81
71
 
82
- See spec/README for instructions on running specs.
72
+ Problems or questions? Add an {issue on GitHub}[https://github.com/ryanb/populator/issues] or fork the project and send a pull request.
83
73
 
84
- This project can be found on github at the following URL.
74
+ See {spec/README}[https://github.com/ryanb/populator/blob/master/spec/README.rdoc] for instructions on running specs.
85
75
 
86
- http://github.com/ryanb/populator
87
76
 
88
- If you find a bug, please send me a message on GitHub.
77
+ == Special Thanks
89
78
 
90
- If you would like to contribute to this project, please fork the
91
- repository and send me a pull request.
79
+ Special thanks to Zach Dennis for his ar-extensions gem which some of this code is based on. Also many thanks to the contributors[https://github.com/ryanb/populator/contributors]. See the CHANGELOG[https://github.com/ryanb/populator/blob/master/CHANGELOG.rdoc] for the full list.
data/Rakefile CHANGED
@@ -1,15 +1,25 @@
1
1
  require 'rubygems'
2
2
  require 'rake'
3
- require 'echoe'
4
-
5
- Echoe.new('populator', '0.2.5') do |p|
6
- p.summary = "Mass populate an Active Record database."
7
- p.description = "Mass populate an Active Record database."
8
- p.url = "http://github.com/ryanb/populator"
9
- p.author = 'Ryan Bates'
10
- p.email = "ryan (at) railscasts (dot) com"
11
- p.ignore_pattern = ["script/*", "**/*.sqlite3", "tmp/*"]
12
- p.development_dependencies = []
13
- end
3
+ require 'rspec/core/rake_task'
4
+ require 'yaml'
5
+
6
+ ADAPTERS = YAML.load(File.read(File.dirname(__FILE__) + "/spec/database.yml")).keys
7
+
8
+ desc "Run specs under all supported databases"
9
+ task :spec => ADAPTERS.map { |a| "spec:#{a}" }
10
+ task :default => :spec
14
11
 
15
- Dir["#{File.dirname(__FILE__)}/tasks/*.rake"].sort.each { |ext| load ext }
12
+ namespace :spec do
13
+ ADAPTERS.each do |adapter|
14
+ namespace :prepare do
15
+ task adapter do
16
+ ENV["POPULATOR_ADAPTER"] = adapter
17
+ end
18
+ end
19
+
20
+ desc "Run specs under #{adapter}"
21
+ RSpec::Core::RakeTask.new(adapter => "spec:prepare:#{adapter}") do |t|
22
+ t.verbose = false
23
+ end
24
+ end
25
+ end
@@ -6,6 +6,8 @@ require 'populator/random'
6
6
 
7
7
  require 'populator/adapters/abstract'
8
8
  require 'populator/adapters/sqlite'
9
+ require 'populator/adapters/oracle'
10
+ require 'populator/adapters/postgresql'
9
11
 
10
12
  # Populator is made up of several parts. To start, see Populator::ModelAdditions.
11
13
  module Populator
@@ -5,7 +5,7 @@ module Populator
5
5
  def execute_batch(sql, name = nil)
6
6
  raise NotImplementedError, "execute_batch is an abstract method"
7
7
  end
8
-
8
+
9
9
  def populate(table, columns, rows, name = nil)
10
10
  execute("INSERT INTO #{table} #{columns} VALUES #{rows.join(', ')}", name)
11
11
  end
@@ -0,0 +1,26 @@
1
+ module Populator
2
+ module Adapters
3
+ module Oracle
4
+
5
+ # Executes SQL statements one at a time.
6
+
7
+ def populate(table, columns, rows, name = nil)
8
+ rows.each do |row|
9
+ sql = "INSERT INTO #{table} #{columns} VALUES #{row}"
10
+ log(sql, name) do
11
+ @connection.exec(sql)
12
+ end
13
+ end
14
+ end
15
+
16
+ end
17
+ end
18
+ end
19
+
20
+ module ActiveRecord # :nodoc: all
21
+ module ConnectionAdapters
22
+ class OracleAdapter < ActiveRecord::ConnectionAdapters::AbstractAdapter
23
+ include Populator::Adapters::Oracle
24
+ end
25
+ end
26
+ end
@@ -0,0 +1,22 @@
1
+ module Populator
2
+ module Adapters
3
+ module Postgresql
4
+ def populate(table, columns, rows, name = nil)
5
+ queries = []
6
+ rows.each do |row|
7
+ row.gsub!(/^\(\d{1,}/, "(DEFAULT")
8
+ queries << "INSERT INTO #{table} #{columns} VALUES #{row}"
9
+ end
10
+ execute(queries.join("; "), name)
11
+ end
12
+ end
13
+ end
14
+ end
15
+
16
+ module ActiveRecord # :nodoc: all
17
+ module ConnectionAdapters
18
+ class PostgreSQLAdapter < ActiveRecord::ConnectionAdapters::AbstractAdapter
19
+ include Populator::Adapters::Postgresql
20
+ end
21
+ end
22
+ end
@@ -3,13 +3,11 @@ module Populator
3
3
  module Sqlite
4
4
  # Executes multiple SQL statements in one query when joined with ";"
5
5
  def execute_batch(sql, name = nil)
6
- catch_schema_changes do
7
- log(sql, name) do
8
- @connection.transaction { |db| db.execute_batch(sql) }
9
- end
6
+ log(sql, name) do
7
+ @connection.transaction { |db| db.execute_batch(sql) }
10
8
  end
11
9
  end
12
-
10
+
13
11
  def populate(table, columns, rows, name = nil)
14
12
  queries = []
15
13
  rows.each do |row|
@@ -1,90 +1,90 @@
1
- module Populator
2
- # Builds multiple Populator::Record instances and saves them to the database
3
- class Factory
4
- DEFAULT_RECORDS_PER_QUERY = 1000
5
-
6
- @factories = {}
7
- @depth = 0
8
-
9
- # Fetches the factory dedicated to a given model class. You should always use this
10
- # method instead of instatiating a factory directly so that a single factory is
11
- # shared on multiple calls.
12
- def self.for_model(model_class)
13
- @factories[model_class] ||= new(model_class)
14
- end
15
-
16
- # Find all remaining factories and call save_records on them.
17
- def self.save_remaining_records
18
- @factories.values.each do |factory|
19
- factory.save_records
20
- end
21
- @factories = {}
22
- end
23
-
24
- # Keep track of nested factory calls so we can save the remaining records once we
25
- # are done with the base factory. This makes Populator more efficient when nesting
26
- # factories.
27
- def self.remember_depth
28
- @depth += 1
29
- yield
30
- @depth -= 1
31
- save_remaining_records if @depth.zero?
32
- end
33
-
34
- # Use for_model instead of instatiating a record directly.
35
- def initialize(model_class)
36
- @model_class = model_class
37
- @records = []
38
- end
39
-
40
- # Entry method for building records. Delegates to build_records after remember_depth.
41
- def populate(amount, options = {}, &block)
42
- self.class.remember_depth do
43
- build_records(Populator.interpret_value(amount), options[:per_query] || DEFAULT_RECORDS_PER_QUERY, &block)
44
- end
45
- end
46
-
47
- # Builds multiple Populator::Record instances and calls save_records them when
48
- # :per_query limit option is reached.
49
- def build_records(amount, per_query, &block)
50
- amount.times do
51
- record = Record.new(@model_class, last_id_in_database + @records.size + 1)
52
- @records << record
53
- block.call(record) if block
54
- save_records if @records.size >= per_query
55
- end
56
- end
57
-
58
- # Saves the records to the database by calling populate on the current database adapter.
59
- def save_records
60
- unless @records.empty?
61
- @model_class.connection.populate(@model_class.quoted_table_name, columns_sql, rows_sql_arr, "#{@model_class.name} Populate")
62
- @last_id_in_database = @records.last.id
63
- @records.clear
64
- end
65
- end
66
-
67
- private
68
-
69
- def quoted_column_names
70
- @model_class.column_names.map do |column_name|
71
- @model_class.connection.quote_column_name(column_name)
72
- end
73
- end
74
-
75
- def last_id_in_database
76
- @last_id_in_database ||= @model_class.connection.select_value("SELECT id FROM #{@model_class.quoted_table_name} ORDER BY id DESC", "#{@model_class.name} Last ID").to_i
77
- end
78
-
79
- def columns_sql
80
- "(#{quoted_column_names.join(', ')})"
81
- end
82
-
83
- def rows_sql_arr
84
- @records.map do |record|
85
- quoted_attributes = record.attribute_values.map { |v| @model_class.sanitize(v) }
86
- "(#{quoted_attributes.join(', ')})"
87
- end
88
- end
89
- end
90
- end
1
+ module Populator
2
+ # Builds multiple Populator::Record instances and saves them to the database
3
+ class Factory
4
+ DEFAULT_RECORDS_PER_QUERY = 1000
5
+
6
+ @factories = {}
7
+ @depth = 0
8
+
9
+ # Fetches the factory dedicated to a given model class. You should always use this
10
+ # method instead of instatiating a factory directly so that a single factory is
11
+ # shared on multiple calls.
12
+ def self.for_model(model_class)
13
+ @factories[model_class] ||= new(model_class)
14
+ end
15
+
16
+ # Find all remaining factories and call save_records on them.
17
+ def self.save_remaining_records
18
+ @factories.values.each do |factory|
19
+ factory.save_records
20
+ end
21
+ @factories = {}
22
+ end
23
+
24
+ # Keep track of nested factory calls so we can save the remaining records once we
25
+ # are done with the base factory. This makes Populator more efficient when nesting
26
+ # factories.
27
+ def self.remember_depth
28
+ @depth += 1
29
+ yield
30
+ @depth -= 1
31
+ save_remaining_records if @depth.zero?
32
+ end
33
+
34
+ # Use for_model instead of instatiating a record directly.
35
+ def initialize(model_class)
36
+ @model_class = model_class
37
+ @records = []
38
+ end
39
+
40
+ # Entry method for building records. Delegates to build_records after remember_depth.
41
+ def populate(amount, options = {}, &block)
42
+ self.class.remember_depth do
43
+ build_records(Populator.interpret_value(amount), options[:per_query] || DEFAULT_RECORDS_PER_QUERY, &block)
44
+ end
45
+ end
46
+
47
+ # Builds multiple Populator::Record instances and calls save_records them when
48
+ # :per_query limit option is reached.
49
+ def build_records(amount, per_query, &block)
50
+ amount.times do
51
+ record = Record.new(@model_class, last_id_in_database + @records.size + 1)
52
+ @records << record
53
+ block.call(record) if block
54
+ save_records if @records.size >= per_query
55
+ end
56
+ end
57
+
58
+ # Saves the records to the database by calling populate on the current database adapter.
59
+ def save_records
60
+ unless @records.empty?
61
+ @model_class.connection.populate(@model_class.quoted_table_name, columns_sql, rows_sql_arr, "#{@model_class.name} Populate")
62
+ @last_id_in_database = @records.last.id
63
+ @records.clear
64
+ end
65
+ end
66
+
67
+ private
68
+
69
+ def quoted_column_names
70
+ @model_class.column_names.map do |column_name|
71
+ @model_class.connection.quote_column_name(column_name)
72
+ end
73
+ end
74
+
75
+ def last_id_in_database
76
+ @last_id_in_database ||= @model_class.connection.select_value("SELECT #{@model_class.primary_key} FROM #{@model_class.quoted_table_name} ORDER BY #{@model_class.primary_key} DESC", "#{@model_class.name} Last ID").to_i
77
+ end
78
+
79
+ def columns_sql
80
+ "(#{quoted_column_names.join(', ')})"
81
+ end
82
+
83
+ def rows_sql_arr
84
+ @records.map do |record|
85
+ quoted_attributes = record.attribute_values.map { |v| @model_class.sanitize(v) }
86
+ "(#{quoted_attributes.join(', ')})"
87
+ end
88
+ end
89
+ end
90
+ end