mbleigh-seed-fu 1.0.0 → 1.2.1

data/README.rdoc

@@ -0,0 +1,149 @@
+= Seed Fu
+
+Seed Fu is an attempt to once and for all solve the problem of inserting and
+maintaining seed data in a database. It uses a variety of techniques gathered
+from various places around the web and combines them to create what is 
+hopefully the most robust seed data system around.
+
+
+== Simple Usage
+
+Seed data is taken from the <tt>db/fixtures</tt> directory. Simply make descriptive .rb
+files in that directory (they can be named anything, but users.rb for the User
+model seed data makes sense, etc.). These scripts will be run whenever the
+db:seed rake task is called, and in order (you can use <tt>00_first.rb</tt>,
+<tt>00_second.rb</tt>, etc). You can put arbitrary Ruby code in these files,
+but remember that it will be executed every time the rake task is called, so
+it needs to be runnable multiple times on the same database.
+
+You can also have environment-specific seed data placed in 
+<tt>db/fixtures/ENVIRONMENT</tt> that will only be loaded if that is the current
+environment.
+
+Let's say we want to populate a few default users. In <tt>db/fixtures/users.rb</tt> we
+write the following code:
+
+    User.seed(:login, :email) do |s|
+      s.login = "bob"
+      s.email = "bob@bobson.com"
+      s.first_name = "Bob"
+      s.last_name = "Bobson"
+    end
+    
+    User.seed(:login, :email) do |s|
+      s.login = "bob"
+      s.email = "bob@stevenson.com"
+      s.first_name = "Bob"
+      s.last_name = "Stevenson"
+    end
+
+That's all you have to do! You will now have two users created in the system
+and you can change their first and last names in the users.rb file and it will
+be updated as such.
+
+The arguments passed to the <tt><ActiveRecord>.seed</tt> method are the constraining
+attributes: these must remain unchanged between db:seed calls to avoid data
+duplication. By default, seed data will constrain to the id like so:
+
+    Category.seed do |s|
+      s.id = 1
+      s.name = "Buttons"
+    end
+    
+    Category.seed do |s|
+      s.id = 2
+      s.name = "Bobbins"
+      s.parent_id = 1
+    end
+    
+Note that any constraints that are passed in must be present in the subsequent
+seed data setting.
+
+== Seed-many Usage
+
+The default <tt>.seed` syntax is very verbose.  An alternative is the `.seed_many</tt> syntax.
+Look at this refactoring of the first seed usage example above:
+
+    User.seed_many(:login, :email, [
+      { :login => "bob", :email => "bob@bobson.com",    :first_name => "Bob", :last_name = "Bobson" },
+      { :login => "bob", :email => "bob@stevenson.com", :first_name => "Bob", :last_name = "Stevenson" }
+    ])
+
+Not as pretty, but much more concise.
+
+== Handling Large SeedFu Files
+
+Seed files can be huge.  To handle large files (over a million rows), try these tricks:
+
+* Gzip your fixtures.  Seed Fu will read .rb.gz files happily.  <tt>gzip -9</tt> gives the
+  best compression, and with Seed Fu's repetitive syntax, a 160M file can shrink to 16M.
+* Add lines reading <tt># BREAK EVAL</tt> in your big fixtures, and Seed Fu will avoid loading
+  the whole file into memory.  If you use SeedFu::Writer, these breaks are built into
+  your generated fixtures.
+* Load a single fixture with the <tt>SEED</tt> environment varialble:
+  <tt>SEED=cities,states rake db:seed > seed_log`.  Each argument to `SEED</tt> is used as a
+  regex to filter fixtures by filename.
+
+== Generating SeedFu Files
+
+Say you have a CSV you need to massage and store as seed files.  You can create
+an import script using SeedFu::Writer.
+
+    #!/usr/bin/env ruby
+    #
+    # This is: script/generate_cities_seed_from_csv
+    #
+    require 'rubygems'
+    require 'fastercsv'
+    require File.join( File.dirname(__FILE__), '..', 'vendor/plugins/seed-fu/lib/seed-fu/writer' )
+
+    # Maybe SEEF_FILE could be $stdout, hm.
+    #
+    CITY_CSV  = File.join( File.dirname(__FILE__), '..', 'city.csv' )
+    SEED_FILE = File.join( File.dirname(__FILE__), '..', 'db', 'fixtures', 'cities.rb' )
+
+    # Create a seed_writer, walk the CSV, add to the file.
+    #
+
+    seed_writer = SeedFu::Writer::SeedMany.new(
+      :seed_file  => SEED_FILE,
+      :seed_model => 'City',
+      :seed_by    => [ :city, :state ]
+    )
+
+    FasterCSV.foreach( CITY_CSV,
+      :return_headers => false,
+      :headers => :first_row
+    ) do |row|
+
+      # Skip all but the US
+      #
+      next unless row['country_code'] == 'US'
+
+      unless us_state
+        puts "No State Match for #{row['region_name']}"
+        next
+      end
+
+      # Write the seed
+      #
+      seed_writer.add_seed({
+        :zip => row['zipcode'],
+        :state => row['state'],
+        :city => row['city'],
+        :latitude => row['latitude'],
+        :longitude => row['longitude']
+      })
+
+    end
+
+    seed_writer.finish
+
+There is also a SeedFu::Writer::Seed in case you prefere the seed()
+syntax over the seen_many() syntax.  Easy-peasy.
+
+== Contributors
+
+* Thanks to Matthew Beale for his great work in adding the writer, making it faster and better.
+
+Copyright (c) 2008-2009 Michael Bleigh released under the MIT license

data/Rakefile

@@ -2,5 +2,22 @@ require 'rake'
 require 'rake/testtask'
 require 'rake/rdoctask'
 
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gemspec|
+    gemspec.name = "seed-fu"
+    gemspec.summary = "Allows easier database seeding of tables in Rails."
+    gemspec.email = "michael@intridea.com"
+    gemspec.homepage = "http://github.com/mblegih/seed-fu"
+    gemspec.description = "Seed Fu is an attempt to once and for all solve the problem of inserting and maintaining seed data in a database. It uses a variety of techniques gathered from various places around the web and combines them to create what is hopefully the most robust seed data system around."
+    gemspec.authors = ["Michael Bleigh"]
+    gemspec.add_dependency 'rails', '>= 2.1'
+    gemspec.files = FileList["[A-Z]*", "{lib,spec,rails}/**/*"] - FileList["**/*.log"]
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+
 desc 'Default: run specs.'
 task :default => :spec

data/VERSION

@@ -0,0 +1 @@
+1.2.1

data/lib/seed-fu.rb

@@ -25,24 +25,21 @@ module SeedFu
       end
     end
 
-    def set_attribute(name, value)
-      @data[name.to_sym] = value
-    end
-
     def plant! insert_only=false
       record = get
       return if !record.new_record? and insert_only
       @data.each do |k, v|
         record.send("#{k}=", v)
       end
-      record.save!
+      raise "Error Saving: #{record.inspect}" unless record.save(false)
       puts " - #{@model_class} #{condition_hash.inspect}"      
       record
     end
 
     def method_missing(method_name, *args) #:nodoc:
-      if (match = method_name.to_s.match(/(.*)=$/)) && args.size == 1
-        set_attribute(match[1], args.first)
+      if args.size == 1 and (match = method_name.to_s.match(/(.*)=$/))
+        self.class.class_eval "def #{method_name} arg; @data[:#{match[1]}] = arg; end"
+        send(method_name, args[0])
       else
         super
       end
@@ -61,7 +58,7 @@ module SeedFu
     end
 
     def condition_hash
-      @data.reject{|a,v| !@constraints.include?(a)}
+      @constraints.inject({}) {|a,c| a[c] = @data[c]; a }
     end
   end
 end
@@ -87,4 +84,4 @@ class ActiveRecord::Base
       end
     end
   end
-end
+end

data/lib/seed-fu/writer.rb

@@ -0,0 +1,3 @@
+require File.join( File.dirname(__FILE__), 'writer', 'abstract' )
+require File.join( File.dirname(__FILE__), 'writer', 'seed' )
+require File.join( File.dirname(__FILE__), 'writer', 'seed_many' )

data/lib/seed-fu/writer/abstract.rb

@@ -0,0 +1,58 @@
+module SeedFu
+  
+  module Writer
+
+    class Abstract
+      attr_accessor :seed_handle, :config, :number_of_seeds
+
+      def initialize(options={})
+        self.config = options
+        self.number_of_seeds = 0
+
+        self.seed_handle = File.new(self.config[:seed_file], 'w')
+
+        write_header
+      end
+      
+      def header
+        <<-END
+# DO NOT MODIFY THIS FILE, it was auto-generated.
+# 
+# Date: #{DateTime.now}
+# Using #{self.class} to seed #{config[:seed_model]}
+# Written with the command:
+#
+#   #{$0} #{$*.join}
+#
+        END
+      end
+
+      def footer
+        <<-END
+# End auto-generated file.
+        END
+      end
+
+      def add_seed(hash)
+        $stdout.puts "Added #{hash.inspect}" unless config[:quiet]
+        self.number_of_seeds += 1
+      end
+
+      def write_header
+        seed_handle.syswrite header
+      end
+
+      def write_footer
+        seed_handle.syswrite footer
+      end
+
+      def finish
+        write_footer
+        seed_handle.close
+      end
+
+    end
+  
+  end
+
+end

data/lib/seed-fu/writer/seed.rb

@@ -0,0 +1,26 @@
+module SeedFu
+  
+  module Writer
+
+    class Seed < Abstract
+
+      # This method uses the :seed_by set earlier.
+      #
+      def add_seed(hash, seed_by=nil)
+        seed_by ||= config[:seed_by]
+        seed_handle.syswrite( <<-END
+#{config[:seed_model]}.seed(#{seed_by.collect{|s| ":#{s}"}.join(',')}) { |s|
+#{hash.collect{|k,v| "  s.#{k} = '#{v.to_s.gsub("'", "\'")}'\n"}.join}}
+        END
+        )
+        super(hash)
+        if chunk_this_seed?
+          seed_handle.syswrite "# BREAK EVAL\n"
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/seed-fu/writer/seed_many.rb

@@ -0,0 +1,52 @@
+module SeedFu
+  
+  module Writer
+
+    class SeedMany < Abstract
+
+      def seed_many_header
+        "#{config[:seed_model]}.seed_many(#{config[:seed_by].collect{|s| ":#{s}"}.join(',')},["
+      end
+      
+      def seed_many_footer
+        "\n])\n"
+      end
+
+      # Chunk in groups of 100 for performance
+      #
+      def chunk_this_seed?
+        0 == (self.number_of_seeds % (config[:chunk_size] || 100))
+      end
+
+      def add_seed(hash)
+        seed_handle.syswrite( (<<-END
+#{',' unless self.number_of_seeds == 0 or chunk_this_seed?}
+  { #{hash.collect{|k,v| ":#{k} => '#{v.to_s.gsub("'", "\'")}'"}.join(', ')} }
+        END
+        ).chomp )
+        super(hash)
+
+        if chunk_this_seed?
+          seed_handle.syswrite(
+            self.seed_many_footer +
+            "# BREAK EVAL\n" +
+            self.seed_many_header
+          )
+        end
+      end
+
+      def write_header
+        super
+        seed_handle.syswrite self.seed_many_header
+      end
+
+      def write_footer
+        seed_handle.syswrite self.seed_many_footer
+        super
+      end
+
+    end
+
+  end
+
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: mbleigh-seed-fu
 version: !ruby/object:Gem::Version 
-  version: 1.0.0
+  version: 1.2.1
 platform: ruby
 authors: 
 - Michael Bleigh
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2008-08-16 00:00:00 -07:00
+date: 2009-09-23 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -29,25 +29,26 @@ executables: []
 extensions: []
 
 extra_rdoc_files: 
-- README
+- README.rdoc
 files: 
-- README
+- README.rdoc
 - Rakefile
-- init.rb
+- VERSION
 - lib/seed-fu.rb
-- lib/autotest/discover.rb
+- lib/seed-fu/writer.rb
+- lib/seed-fu/writer/abstract.rb
+- lib/seed-fu/writer/seed.rb
+- lib/seed-fu/writer/seed_many.rb
 - rails/init.rb
-- seed-fu.gemspec
 - spec/schema.rb
 - spec/seed_fu_spec.rb
 - spec/spec_helper.rb
-- tasks/seed_fu_tasks.rake
-has_rdoc: true
-homepage: http://github.com/mbleigh/seed-fu
+has_rdoc: false
+homepage: http://github.com/mblegih/seed-fu
+licenses: 
 post_install_message: 
 rdoc_options: 
-- --main
-- README
+- --charset=UTF-8
 require_paths: 
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement 
@@ -65,9 +66,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 
 rubyforge_project: 
-rubygems_version: 1.2.0
+rubygems_version: 1.3.5
 signing_key: 
-specification_version: 2
-summary: Allows easier database seeding of tables.
-test_files: []
-
+specification_version: 3
+summary: Allows easier database seeding of tables in Rails.
+test_files: 
+- spec/schema.rb
+- spec/seed_fu_spec.rb
+- spec/spec_helper.rb

data/README

@@ -1,66 +0,0 @@
-Seed Fu
-=======
-
-Seed Fu is an attempt to once and for all solve the problem of inserting and
-maintaining seed data in a database. It uses a variety of techniques gathered
-from various places around the web and combines them to create what is 
-hopefully the most robust seed data system around.
-
-If you have suggestions or come across problems, please report them on
-the Lighthouse project for Seed Fu:
-http://mbleigh.lighthouseapp.com/projects/10223-seed-fu/overview
-
-Usage
-=======
-
-Seed data is taken from the db/fixtures directory. Simply make descriptive .rb
-files in that directory (they can be named anything, but users.rb for the User
-model seed data makes sense, etc.). These scripts will be run whenever the
-db:seed rake task is called. You can put arbitrary Ruby code in these files,
-but remember that it will be executed every time the rake task is called, so
-it needs to be runnable multiple times on the same database.
-
-You can also have environment-specific seed data placed in 
-db/fixtures/ENVIRONMENT that will only be loaded if that is the current
-environment.
-
-Let's say we want to populate a few default users. In db/fixtures/users.rb we
-write the following code:
-
-    User.seed(:login, :email) do |s|
-      s.login = "bob"
-      s.email = "bob@bobson.com"
-      s.first_name = "Bob"
-      s.last_name = "Bobson"
-    end
-    
-    User.seed(:login, :email) do |s|
-      s.login = "bob"
-      s.email = "bob@stevenson.com"
-      s.first_name = "Bob"
-      s.last_name = "Stevenson"
-    end
-
-That's all you have to do! You will now have two users created in the system
-and you can change their first and last names in the users.rb file and it will
-be updated as such.
-
-The arguments passed to the <ActiveRecord>.seed method are the constraining
-attributes: these must remain unchanged between db:seed calls to avoid data
-duplication. By default, seed data will constrain to the id like so:
-
-    Category.seed do |s|
-      s.id = 1
-      s.name = "Buttons"
-    end
-    
-    Category.seed do |s|
-      s.id = 2
-      s.name = "Bobbins"
-      s.parent_id = 1
-    end
-    
-Note that any constraints that are passed in must be present in the subsequent
-seed data setting.
-
-Copyright (c) 2008 Michael Bleigh, released under the MIT license

data/init.rb

@@ -1 +0,0 @@
-require File.dirname(__FILE__) + "/rails/init"

data/seed-fu.gemspec

@@ -1,31 +0,0 @@
-Gem::Specification.new do |s|
-  s.name = 'seed-fu'
-  s.version = '1.0.0'
-  s.date = '2008-08-16'
-  
-  s.summary = "Allows easier database seeding of tables."
-  s.description = "Seed Fu is an attempt to once and for all solve the problem of inserting and maintaining seed data in a database. It uses a variety of techniques gathered from various places around the web and combines them to create what is hopefully the most robust seed data system around."
-  
-  s.authors = ["Michael Bleigh"]
-  s.email = "michael@intridea.com"
-  s.homepage = 'http://github.com/mbleigh/seed-fu'
-  
-  s.has_rdoc = true
-  s.rdoc_options = ["--main", "README"]
-  s.extra_rdoc_files = ["README"]
-
-  s.add_dependency 'rails', ['>= 2.1']
-  
-  s.files = ["README",
-             "Rakefile",
-             "init.rb",
-             "lib/seed-fu.rb",
-             "lib/autotest/discover.rb",
-             "rails/init.rb",
-             "seed-fu.gemspec",
-             "spec/schema.rb",
-             "spec/seed_fu_spec.rb",
-             "spec/spec_helper.rb",
-             "tasks/seed_fu_tasks.rake"]
-
-end

data/tasks/seed_fu_tasks.rake

@@ -1,44 +0,0 @@
-namespace :db do
-  desc <<-EOS
-    Loads seed data for the current environment. It will look for
-    ruby seed files in <RAILS_ROOT>/db/fixtures/ and 
-    <RAILS_ROOT>/db/fixtures/<RAILS_ENV>/.
-
-    By default it will load any ruby files found. You can filter the files
-    loaded by passing in the SEED environment variable with a comma-delimited
-    list of patterns to include. Any files not matching the pattern will
-    not be loaded.
-    
-    You can also change the directory where seed files are looked for
-    with the FIXTURE_PATH environment variable. 
-    
-    Examples:
-      # default, to load all seed files for the current environment
-      rake db:seed
-      
-      # to load seed files matching orders or customers
-      rake db:seed SEED=orders,customers
-      
-      # to load files from RAILS_ROOT/features/fixtures
-      rake db:seed FIXTURE_PATH=features/fixtures 
-  EOS
-  task :seed => :environment do
-    fixture_path = ENV["FIXTURE_PATH"] ? ENV["FIXTURE_PATH"] : "db/fixtures"
-
-    global_seed_files = Dir[File.join(RAILS_ROOT, fixture_path, '*.rb')].sort
-    env_specific_seed_files = Dir[File.join(RAILS_ROOT, fixture_path, RAILS_ENV, '*.rb')]
-    potential_seed_files = (global_seed_files + env_specific_seed_files).uniq
-    
-    if ENV["SEED"]
-      filter = ENV["SEED"].gsub(/,/, "|")
-      potential_seed_files.reject!{ |file| true unless file =~ /#{filter}/ }
-      puts "\n == Filtering seed files against regexp: #{filter}"
-    end
-
-    potential_seed_files.each do |file|
-      pretty_name = file.sub("#{RAILS_ROOT}/", "")
-      puts "\n== Seed from #{pretty_name} " + ("=" * (60 - (17 + File.split(file).last.length)))
-      load file
-    end
-  end
-end