couchpopulator 0.1.0 → 0.2.0

data/README.md

@@ -15,7 +15,7 @@ This project is in a very early state. I'm sure it has some serious bugs and it'
 
 # Getting Started
 
-## Gem
+## Installation
 
     sudo gem install couchpopulator
 
@@ -27,18 +27,25 @@ This project is in a very early state. I'm sure it has some serious bugs and it'
     rake build
       
 ## Getting help
+CouchPopulator tries to give good help on command line options by using:
 
     couchpopulator --help
+    
+To get command line options to a specific execution engine, simply use:
+
+    couchpopulator [EXECUTOR] --help
 
 ## Custom Generators
 Custom generators only need to implement one method. Have a look:
 
-    module Generators
-      class Example
-        class << self
-          def generate(count)
-            # ...heavy generating action goes here...
-            # return array of hashes (documents)
+    module CouchPopulator
+      module Generators
+        class Example
+          class << self
+            def generate(count)
+              # ...heavy generating action goes here...
+              # return array of hashes (documents)
+            end
           end
         end
       end
@@ -48,14 +55,39 @@ generate(count) should return an array of documents. Each document should be an
 
 
 ## Custom Execution Engines
-Custom execute engines need to implement two methods `troll_options` and `execute`. See `executors/standard.rb` for an example.
+Custom execute engines need to implement two methods `command_line_options` and `execute`. See `executors/standard.rb` for an example.
+
+# Using the CouchPopulator API
+This is the very first version of the CouchPopulator API (introduced in v0.2.0). The interface is still very ugly and will change significantly in the future.
+
+The options for `CouchPopulator::Base`'s initializer are the same as the command line options. No rules without exceptions:
+
+* `:logger` needs to be set to a instance of `CouchPopulator::Logger`
+* `:generator_klass` needs to be set to the generator constant `CouchPopulator::Initializer.generator()` tries to load it
+* `:executor_klass` needs to be set to the generator constant `CouchPopulator::Initializer.executor()` tries to load it
+
+Example:
+
+    require 'rubygems'
+    require 'couchpopulator'
+
+    options = { :logger => CouchPopulator::Logger.new,
+                :generator_klass => CouchPopulator::Initializer.generator('example'),
+                :executor_klass => CouchPopulator::Initializer.executor('standard'),
+                :couch => 'http://localhost:5984/test',
+                :docs_per_chunk => 1,
+                :rounds => 1,
+                :concurrent_inserts => 1 }
 
+    CouchPopulator::Base.new(options).populate
 
 # TODO
+- make the API suck less
+- Add support for using a configuration YAML
 - Find out the best strategies for inserting docs to CouchDB and provide execution engines for different approches
 - Implement some more features, like dumping-options for generated documents or load dumped JSON docs to CouchDB
 - Think about a test suite and implement it
-- hunting bugs, make it cleaner, make a gem, ...
+- hunting bugs, make it cleaner
 
 
 

data/executors/standard.rb

@@ -1,72 +1,83 @@
-module Executors
-  class Standard
-    def initialize(opts={})
-      @opts = opts.merge(command_line_options)
-    end
+module CouchPopulator
+  module Executors
+    class Standard
+      def initialize(opts={})
+        @options = self.class.defaults.merge(opts)
+      end
+
+      def self.defaults
+        @defaults ||= {
+          :docs_per_chunk => 1000,
+          :concurrent_inserts => 5,
+          :rounds => 1
+        }
+      end
 
-    def command_line_options
-      help = StringIO.new
+      def self.command_line_options
+        help = StringIO.new
 
-      opts = Trollop.options do
-        version "StandardExecutor v0.1 (c) Sebastian Cohnen, 2009"
-        banner <<-BANNER
-This is the StandardExecutor
-        BANNER
-        opt :docs_per_chunk, "Number of docs per chunk", :default => 2000
-        opt :concurrent_inserts, "Number of concurrent inserts", :default => 5
-        opt :rounds, "Number of rounds", :default => 2
-        opt :preflight, "Generate the docs, but don't write to couch. Use with ", :default => false
-        opt :help, "Show this message"
+        defaults = self.defaults
 
-        educate(help)
-      end
+        opts = Trollop.options do
+          version "StandardExecutor v0.1 (c) Sebastian Cohnen, 2009"
+          banner <<-BANNER
+  This is the StandardExecutor
+          BANNER
+          opt :docs_per_chunk, "Number of docs per chunk", :default => defaults[:docs_per_chunk]
+          opt :concurrent_inserts, "Number of concurrent inserts", :default => defaults[:concurrent_inserts]
+          opt :rounds, "Number of rounds", :default => defaults[:rounds]
+          opt :help, "Show this message"
 
-      if opts[:help]
-        puts help.rewind.read
-        exit
-      else
-        return opts
+          educate(help)
+        end
+
+        if opts[:help]
+          puts help.rewind.read
+          exit
+        else
+          return opts
+        end
       end
-    end
 
-    def execute
-      rounds = @opts[:rounds]
-      docs_per_chunk = @opts[:docs_per_chunk]
-      concurrent_inserts = @opts[:concurrent_inserts]
-      generator = @opts[:generator_klass]  
+      def execute
+        rounds = @options[:rounds]
+        docs_per_chunk = @options[:docs_per_chunk]
+        concurrent_inserts = @options[:concurrent_inserts]
+        generator = @options[:generator_klass]
 
-      log = @opts[:logger]
-      log << "CouchPopulator's default execution engine has been started."
-      log << "Using #{generator.to_s} for generating the documents."
+        log = @options[:logger]
+        log << "CouchPopulator's default execution engine has been started."
+        log << "Using #{generator.to_s} for generating the documents."
 
-      total_docs = docs_per_chunk * concurrent_inserts * rounds
-      log << "Going to insert #{total_docs} generated docs into #{@opts[:couch_url]}"
-      log << "Using #{rounds} rounds of #{concurrent_inserts} concurrent inserts with #{docs_per_chunk} docs each"
+        total_docs = docs_per_chunk * concurrent_inserts * rounds
+        log << "Going to insert #{total_docs} generated docs into #{@options[:couch_url]}"
+        log << "Using #{rounds} rounds of #{concurrent_inserts} concurrent inserts with #{docs_per_chunk} docs each"
 
-      start_time = Time.now
+        start_time = Time.now
 
-      rounds.times do |round|
-        log << "Starting with round #{round + 1}"
-        concurrent_inserts.times do
-          fork do
-            # generate payload for bulk_doc
-            payload = ({"docs" => generator.generate(docs_per_chunk)}).to_json
+        rounds.times do |round|
+          log << "Starting with round #{round + 1}"
+          concurrent_inserts.times do
+            fork do
+              # generate payload for bulk_doc
+              payload = JSON.generate({"docs" => generator.generate(docs_per_chunk)})
 
-            unless @opts[:generate_only]
-              result = CurlAdapter::Invoker.new(@opts[:couch_url]).post(payload)
-            else
-              log << "Generated chunk..."
-              puts payload
+              unless @options[:generate_only]
+                result = CurlAdapter::Invoker.new(@options[:couch_url]).post(payload)
+              else
+                log << "Generated chunk..."
+                puts payload
+              end
             end
           end
+          concurrent_inserts.times { Process.wait() }
         end
-        concurrent_inserts.times { Process.wait() }
+
+        end_time = Time.now
+        duration = end_time - start_time
+
+        log << "Execution time: #{duration}s, #{@options[:generate_only] ? "generated" : "inserted"} #{total_docs}"
       end
-      
-      end_time = Time.now
-      duration = end_time - start_time
-      
-      log << "Execution time: #{duration}s, inserted #{total_docs}"
     end
   end
 end

data/generators/example.rb

@@ -1,15 +1,17 @@
-module Generators
-  class Example
-    class << self
-      def generate(count)
-        docs = []
-        count.times do
-          docs << {
-            "title"       => "Example",
-            "created_at"  => Time.now - (rand(7) * 60*60*24)
-          }
+module CouchPopulator
+  module Generators
+    class Example
+      class << self
+        def generate(count)
+          docs = []
+          count.times do
+            docs << {
+              "title"       => "Example",
+              "created_at"  => Time.now - (rand(7) * 60*60*24)
+            }
+          end
+          docs
         end
-        docs
       end
     end
   end

data/lib/couchpopulator/base.rb

@@ -1,24 +1,23 @@
 module CouchPopulator
   class Base
-    def initialize(options)
-      @opts = options
-      @opts[:couch_url] = CouchHelper.get_full_couchurl options[:couch] unless options[:couch].nil?
-      @logger = options[:logger]
+    def initialize(options, called_from_command_line = false)
+      @options = options
+
+      @options[:couch_url] = CouchHelper.get_full_couchurl options[:couch] unless @options[:couch].nil?
+      @options.merge!(@options[:executor_klass].command_line_options) if called_from_command_line
     end
 
     def populate
-      @opts[:logger] ||= @logger
-
-      @opts[:database] ||= database
-      @opts[:executor_klass].new(@opts).execute
+      @options[:executor_klass].new(@options).execute
     end
+  end
 
-    def log(message)
-      @logger.log(message)
-    end
+  class CouchPopulatorError < StandardError
+  end
 
-    def database
-      URI.parse(@opts[:couch_url]).path unless @opts[:couch_url].nil?
-    end
+  class GeneratorNotFound < CouchPopulatorError
+  end
+
+  class ExecutorNotFound < CouchPopulatorError
   end
 end

data/lib/couchpopulator/couch_helper.rb

@@ -1,12 +1,14 @@
 module CouchPopulator
-  # Borrowed from Rails
-  # http://github.com/rails/rails/blob/ea0e41d8fa5a132a2d2771e9785833b7663203ac/activesupport/lib/active_support/inflector.rb#L355
   class CouchHelper
     class << self
       def get_full_couchurl(arg)
         arg.match(/^https?:\/\//) ? arg : URI.join('http://127.0.0.1:5984/', arg).to_s
       end
       
+      def get_database_from_couchurl(url)
+        URI.parse(url).path
+      end
+
       def couch_available? (couch_url)
         # TODO this uri-thing is ugly :/
         tmp = URI.parse(couch_url)

data/lib/couchpopulator/initializer.rb

@@ -25,9 +25,23 @@ module CouchPopulator
           end
         end
 
+        # Get the generator_klass and executor_klass
+        generator_klass = begin
+          generator(command_line_options[:generator])
+        rescue CouchPopulator::GeneratorNotFound
+          Trollop.die :generator, "Generator must be set, a valid class-name and respond to generate(n)"
+        end
+
+        executor_klass = begin
+          executor(executor(ARGV.shift || "standard"))
+        rescue CouchPopulator::ExecutorNotFound
+          Trollop.die "Executor must be set and a valid class-name"
+        end
+
+
         # Initialize CouchPopulator
-        options = ({:executor_klass => executor, :generator_klass => generator, :logger => CouchPopulator::Logger.new(command_line_options[:logfile])}).merge(command_line_options)
-        CouchPopulator::Base.new(options).populate
+        options = ({:executor_klass => executor_klass, :generator_klass => generator_klass, :logger => CouchPopulator::Logger.new(command_line_options[:logfile])}).merge(command_line_options)
+        CouchPopulator::Base.new(options, true).populate
       end
 
       # Define some command-line options
@@ -57,35 +71,34 @@ OPTIONS:
         end
       end
 
-      # Get the requested generator or die
-      def generator
+      # Get the requested generator constant or die
+      def generator(generator)
         retried = false
         @generator ||= begin
-            generator_klass = CouchPopulator::MiscHelper.camelize_and_constantize("generators/#{command_line_options[:generator]}")
+            generator_klass = CouchPopulator::MiscHelper.camelize_and_constantize("couch_populator/generators/#{generator}")
           rescue NameError
             begin
-              require File.join(File.dirname(__FILE__), "../../generators/#{command_line_options[:generator]}.rb")
+              require File.join(File.dirname(__FILE__), "../../generators/#{generator}.rb")
             rescue LoadError; end # just catch, do nothing
             retry if (retried = !retried)
           ensure
-            Trollop.die :generator, "Generator must be set, a valid class-name and respond to generate(n)" if generator_klass.nil?
+            raise CouchPopulator::GeneratorNotFound if generator_klass.nil?
             generator_klass
         end
       end
 
-      # Get the exexcutor (defaults to standard) or die
-      def executor
+      # Get the exexcutor constant (defaults to standard) or die
+      def executor(executor)
         retried = false
         @executor ||= begin
-            executor_cmd ||= ARGV.shift || "standard"
-            executor_klass = CouchPopulator::MiscHelper.camelize_and_constantize("executors/#{executor_cmd}")
+            executor_klass = CouchPopulator::MiscHelper.camelize_and_constantize("couch_populator/executors/#{executor}")
           rescue NameError
             begin
-              require File.join(File.dirname(__FILE__), "../../executors/#{executor_cmd}.rb")
+              require File.join(File.dirname(__FILE__), "../../executors/#{executor}.rb")
             rescue NameError, LoadError; end # just catch, do nothing
             retry if (retried = !retried)
           ensure
-            Trollop.die "Executor must be set and a valid class-name" if executor_klass.nil?
+            raise CouchPopulator::ExecutorNotFound if executor_klass.nil?
             executor_klass
           end
       end

data/lib/couchpopulator/misc_helper.rb

@@ -6,6 +6,8 @@ module CouchPopulator
          constantize(camelize(lower_case_and_underscored_word))
       end
 
+      # Borrowed from Rails
+      # http://github.com/rails/rails/blob/ea0e41d8fa5a132a2d2771e9785833b7663203ac/activesupport/lib/active_support/inflector.rb#L355
       def camelize(lower_case_and_underscored_word, first_letter_in_uppercase = true)
         if first_letter_in_uppercase
           lower_case_and_underscored_word.to_s.gsub(/\/(.?)/) { "::#{$1.upcase}" }.gsub(/(?:^|_)(.)/) { $1.upcase }

data/lib/couchpopulator.rb

@@ -5,7 +5,6 @@ require 'uri'
 require 'json/add/rails'
 require 'json/add/core'
 
-require File.join(File.dirname(__FILE__), 'couchpopulator.rb')
 require File.join(File.dirname(__FILE__), 'curl_adapter.rb')
 require File.join(File.dirname(__FILE__), 'generator.rb')
 require File.join(File.dirname(__FILE__), 'logger.rb')

data/lib/curl_adapter.rb

@@ -1,3 +1,4 @@
+# TODO this "thing" realy sucks, make it better! :)
 class CurlAdapter
   class Response
     attr_reader :http_response_code
@@ -29,8 +30,3 @@ class CurlAdapter
     end
   end
 end
-
-# TODO:
-# Keep-Alive mit curl? wäre geil...
-
-

data/lib/logger.rb

@@ -1,6 +1,6 @@
 module CouchPopulator
   class Logger
-    def initialize(logfile)
+    def initialize(logfile='')
       @out = logfile.empty? ? $stdout : File.new(logfile, "a")
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: couchpopulator
 version: !ruby/object:Gem::Version 
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors: 
 - Sebastian Cohnen
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-11-16 00:00:00 +01:00
+date: 2009-11-17 00:00:00 +01:00
 default_executable: couchpopulator
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -32,7 +32,7 @@ dependencies:
       - !ruby/object:Gem::Version 
         version: "1.15"
     version: 
-description: flexible tool for populating CouchDB with generated documents
+description: The idea behind this tool is to provide a framework for populating your CouchDB instances with generated documents. It provides a plug-able system for easy writing own generators. Also the the process, which invokes the generator and manages the insertion to CouchDB, what I call execution engines, are easily exchangeable. The default execution engine uses CouchDB's bulk-docs-API with configurable chunk-size, concurrent inserts and total chunks to insert.
 email: sebastian.cohnen@gmx.net
 executables: 
 - couchpopulator
@@ -86,6 +86,6 @@ rubyforge_project:
 rubygems_version: 1.3.5
 signing_key: 
 specification_version: 3
-summary: flexible tool for populating CouchDB with generated documents
+summary: Flexible tool for populating CouchDB with generated documents
 test_files: []