couchpopulator 0.1.0

data/README.md

@@ -0,0 +1,64 @@
+# CouchPopulator: The Idea
+The idea behind this tool is to provide a framework for populating your [CouchDB][couchdb] instances with generated documents. It provides a plug-able system for easy writing own generators. Also the the process, which invokes the generator and manages the insertion to CouchDB, what I call execution engines, are easily exchangeable. The default execution engine uses CouchDB's [`bulk-docs`][bulk_api]-API with configurable chunk-size, concurrent inserts and total chunks to insert.
+
+# Warning
+This project is in a very early state. I'm sure it has some serious bugs and it's interface and structure for writing own generators and execution engines will definitely change (maybe significantly). I only test it with Ruby 1.8 on OS X 10.6.2.
+
+**Use with caution!**
+
+**BUT**: Please, feel free to comment, fork or fill a ticket with bugs and wishes. You may also drop me a message via GitHub, [@tisba](https://twitter.com/tisba) or at [@couchdb on freenode](irc://irc.freenode.net/couchdb).
+
+
+# Why?
+*"there is tool xy already doing that"* - I don't care (okay, thats not true, I care and I'm always eager to see how others implement stuff). I know that there are some tools providing dumping/loading support for CouchDB. But none is written in ruby and non satisfied my needs (e.g. dynamically generating documents). Nevertheless I wanted to learn how you can write such a tool and get more familiar with CouchDB.
+
+
+# Getting Started
+
+## Gem
+
+    sudo gem install couchpopulator
+
+## Building the gem yourself
+
+    sudo gem install json trollops
+    git clone git@github.com:tisba/couchpopulator.git
+    cd couchpopulator
+    rake build
+      
+## Getting help
+
+    couchpopulator --help
+
+## Custom Generators
+Custom generators only need to implement one method. Have a look:
+
+    module Generators
+      class Example
+        class << self
+          def generate(count)
+            # ...heavy generating action goes here...
+            # return array of hashes (documents)
+          end
+        end
+      end
+    end
+
+generate(count) should return an array of documents. Each document should be an hash that will be encoded in JSON. You can include any object-type you want. CouchPopulator provides some "special" encoding support for Ruby's `time` and `date`. For your own objects you can provide `to_json` and `json_create` used by the [JSON gem][json_gem] to serialise and deserialise it properly.
+
+
+## Custom Execution Engines
+Custom execute engines need to implement two methods `troll_options` and `execute`. See `executors/standard.rb` for an example.
+
+
+# TODO
+- Find out the best strategies for inserting docs to CouchDB and provide execution engines for different approches
+- Implement some more features, like dumping-options for generated documents or load dumped JSON docs to CouchDB
+- Think about a test suite and implement it
+- hunting bugs, make it cleaner, make a gem, ...
+
+
+
+[couchdb]: http://couchdb.apache.org
+[bulk_api]: http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API
+[json_gem]: http://flori.github.com/json/

data/bin/couchpopulator

@@ -0,0 +1,4 @@
+#! /usr/bin/ruby
+require 'couchpopulator'
+
+CouchPopulator::Initializer.run

data/executors/standard.rb

@@ -0,0 +1,72 @@
+module Executors
+  class Standard
+    def initialize(opts={})
+      @opts = opts.merge(command_line_options)
+    end
+
+    def command_line_options
+      help = StringIO.new
+
+      opts = Trollop.options do
+        version "StandardExecutor v0.1 (c) Sebastian Cohnen, 2009"
+        banner <<-BANNER
+This is the StandardExecutor
+        BANNER
+        opt :docs_per_chunk, "Number of docs per chunk", :default => 2000
+        opt :concurrent_inserts, "Number of concurrent inserts", :default => 5
+        opt :rounds, "Number of rounds", :default => 2
+        opt :preflight, "Generate the docs, but don't write to couch. Use with ", :default => false
+        opt :help, "Show this message"
+
+        educate(help)
+      end
+
+      if opts[:help]
+        puts help.rewind.read
+        exit
+      else
+        return opts
+      end
+    end
+
+    def execute
+      rounds = @opts[:rounds]
+      docs_per_chunk = @opts[:docs_per_chunk]
+      concurrent_inserts = @opts[:concurrent_inserts]
+      generator = @opts[:generator_klass]  
+
+      log = @opts[:logger]
+      log << "CouchPopulator's default execution engine has been started."
+      log << "Using #{generator.to_s} for generating the documents."
+
+      total_docs = docs_per_chunk * concurrent_inserts * rounds
+      log << "Going to insert #{total_docs} generated docs into #{@opts[:couch_url]}"
+      log << "Using #{rounds} rounds of #{concurrent_inserts} concurrent inserts with #{docs_per_chunk} docs each"
+
+      start_time = Time.now
+
+      rounds.times do |round|
+        log << "Starting with round #{round + 1}"
+        concurrent_inserts.times do
+          fork do
+            # generate payload for bulk_doc
+            payload = ({"docs" => generator.generate(docs_per_chunk)}).to_json
+
+            unless @opts[:generate_only]
+              result = CurlAdapter::Invoker.new(@opts[:couch_url]).post(payload)
+            else
+              log << "Generated chunk..."
+              puts payload
+            end
+          end
+        end
+        concurrent_inserts.times { Process.wait() }
+      end
+      
+      end_time = Time.now
+      duration = end_time - start_time
+      
+      log << "Execution time: #{duration}s, inserted #{total_docs}"
+    end
+  end
+end

data/generators/example.rb

@@ -0,0 +1,16 @@
+module Generators
+  class Example
+    class << self
+      def generate(count)
+        docs = []
+        count.times do
+          docs << {
+            "title"       => "Example",
+            "created_at"  => Time.now - (rand(7) * 60*60*24)
+          }
+        end
+        docs
+      end
+    end
+  end
+end

data/lib/couchpopulator/base.rb

@@ -0,0 +1,24 @@
+module CouchPopulator
+  class Base
+    def initialize(options)
+      @opts = options
+      @opts[:couch_url] = CouchHelper.get_full_couchurl options[:couch] unless options[:couch].nil?
+      @logger = options[:logger]
+    end
+
+    def populate
+      @opts[:logger] ||= @logger
+
+      @opts[:database] ||= database
+      @opts[:executor_klass].new(@opts).execute
+    end
+
+    def log(message)
+      @logger.log(message)
+    end
+
+    def database
+      URI.parse(@opts[:couch_url]).path unless @opts[:couch_url].nil?
+    end
+  end
+end

data/lib/couchpopulator/couch_helper.rb

@@ -0,0 +1,27 @@
+module CouchPopulator
+  # Borrowed from Rails
+  # http://github.com/rails/rails/blob/ea0e41d8fa5a132a2d2771e9785833b7663203ac/activesupport/lib/active_support/inflector.rb#L355
+  class CouchHelper
+    class << self
+      def get_full_couchurl(arg)
+        arg.match(/^https?:\/\//) ? arg : URI.join('http://127.0.0.1:5984/', arg).to_s
+      end
+      
+      def couch_available? (couch_url)
+        # TODO this uri-thing is ugly :/
+        tmp = URI.parse(couch_url)
+        `curl --fail --request GET #{tmp.scheme}://#{tmp.host}:#{tmp.port} 2> /dev/null`
+        return $?.exitstatus == 0
+      end
+      
+      def database_exists? (db_url)
+        `curl --fail --request GET #{db_url} 2> /dev/null`
+        return $?.exitstatus == 0
+      end
+
+      def create_db(url)
+        !!(JSON.parse(`curl --silent --write-out %{http_code} --request PUT #{url}`))["ok"]
+      end
+    end
+  end
+end

data/lib/couchpopulator/initializer.rb

@@ -0,0 +1,94 @@
+module CouchPopulator
+  class Initializer
+    class << self
+
+      def run
+        # process command line options
+        command_line_options
+
+        # Only check CouchDB-availibilty when needed
+        unless command_line_options[:generate_only]
+          Trollop.die :couch, "You need at least to provide the database's name" if command_line_options[:couch].nil?
+
+          # Build the full CouchDB database url
+          couch_url = CouchHelper.get_full_couchurl(command_line_options[:couch])
+
+          # Check for availabilty of couchdb
+          Trollop.die :couch, "#{couch_url} is not reachable or ressource does not exist" unless CouchHelper.couch_available?(couch_url)
+
+          # create database on demand
+          if command_line_options[:create_db]
+            # TODO needs to be implemented properly
+            # CouchPopulator::CouchHelper.create_db(command_line_options[:couch])
+          else
+            CouchPopulator::CouchHelper.database_exists? couch_url
+          end
+        end
+
+        # Initialize CouchPopulator
+        options = ({:executor_klass => executor, :generator_klass => generator, :logger => CouchPopulator::Logger.new(command_line_options[:logfile])}).merge(command_line_options)
+        CouchPopulator::Base.new(options).populate
+      end
+
+      # Define some command-line options
+      def command_line_options
+        @command_line_options ||= Trollop.options do
+          version "v0.1 (c) Sebastian Cohnen, 2009"
+          banner <<-BANNER
+This is a simple, yet powerfull tool to import large numbers of on-the-fly generated documents into CouchDB.
+It's using concurrency by spawning several curl subprocesses. Documents are generated on-the-fly.
+
+See http://github.com/tisba/couchpopulator for more information.
+
+Usage:
+./couchpopulator [OPTIONS] [executor [EXECUTOR-OPTIONS]]
+
+To see, what options for 'executor' are:
+./couchpopulator executor -h
+
+OPTIONS:
+          BANNER
+          opt :couch, "URL of CouchDB Server. You can also provide the name of the target DB only, http://localhost:5984/ will be prepended automatically", :type => String
+          opt :create_db, "Create DB if needed.", :default => false
+          opt :generator, "Name of the generator-class to use", :default => "Example"
+          opt :generate_only, "Generate the docs, but don't write to couch and stdout them instead", :default => false
+          opt :logfile, "Redirect info/debug output to specified file instead to stdout", :type => String, :default => ""
+          stop_on_unknown
+        end
+      end
+
+      # Get the requested generator or die
+      def generator
+        retried = false
+        @generator ||= begin
+            generator_klass = CouchPopulator::MiscHelper.camelize_and_constantize("generators/#{command_line_options[:generator]}")
+          rescue NameError
+            begin
+              require File.join(File.dirname(__FILE__), "../../generators/#{command_line_options[:generator]}.rb")
+            rescue LoadError; end # just catch, do nothing
+            retry if (retried = !retried)
+          ensure
+            Trollop.die :generator, "Generator must be set, a valid class-name and respond to generate(n)" if generator_klass.nil?
+            generator_klass
+        end
+      end
+
+      # Get the exexcutor (defaults to standard) or die
+      def executor
+        retried = false
+        @executor ||= begin
+            executor_cmd ||= ARGV.shift || "standard"
+            executor_klass = CouchPopulator::MiscHelper.camelize_and_constantize("executors/#{executor_cmd}")
+          rescue NameError
+            begin
+              require File.join(File.dirname(__FILE__), "../../executors/#{executor_cmd}.rb")
+            rescue NameError, LoadError; end # just catch, do nothing
+            retry if (retried = !retried)
+          ensure
+            Trollop.die "Executor must be set and a valid class-name" if executor_klass.nil?
+            executor_klass
+          end
+      end
+    end
+  end
+end

data/lib/couchpopulator/misc_helper.rb

@@ -0,0 +1,61 @@
+module CouchPopulator
+  class MiscHelper
+    class << self
+
+      def camelize_and_constantize(lower_case_and_underscored_word)
+         constantize(camelize(lower_case_and_underscored_word))
+      end
+
+      def camelize(lower_case_and_underscored_word, first_letter_in_uppercase = true)
+        if first_letter_in_uppercase
+          lower_case_and_underscored_word.to_s.gsub(/\/(.?)/) { "::#{$1.upcase}" }.gsub(/(?:^|_)(.)/) { $1.upcase }
+        else
+          lower_case_and_underscored_word.first.downcase + camelize(lower_case_and_underscored_word)[1..-1]
+        end
+      end
+
+      # Ruby 1.9 introduces an inherit argument for Module#const_get and
+      # #const_defined? and changes their default behavior.
+      if Module.method(:const_get).arity == 1
+        # Tries to find a constant with the name specified in the argument string:
+        #
+        #   "Module".constantize     # => Module
+        #   "Test::Unit".constantize # => Test::Unit
+        #
+        # The name is assumed to be the one of a top-level constant, no matter whether
+        # it starts with "::" or not. No lexical context is taken into account:
+        #
+        #   C = 'outside'
+        #   module M
+        #     C = 'inside'
+        #     C               # => 'inside'
+        #     "C".constantize # => 'outside', same as ::C
+        #   end
+        #
+        # NameError is raised when the name is not in CamelCase or the constant is
+        # unknown.
+        def constantize(camel_cased_word)
+          names = camel_cased_word.split('::')
+          names.shift if names.empty? || names.first.empty?
+
+          constant = Object
+          names.each do |name|
+            constant = constant.const_defined?(name) ? constant.const_get(name) : constant.const_missing(name)
+          end
+          constant
+        end
+      else
+        def constantize(camel_cased_word) #:nodoc:
+          names = camel_cased_word.split('::')
+          names.shift if names.empty? || names.first.empty?
+
+          constant = Object
+          names.each do |name|
+            constant = constant.const_get(name, false) || constant.const_missing(name)
+          end
+          constant
+        end
+      end
+    end
+  end
+end

data/lib/couchpopulator.rb

@@ -0,0 +1,13 @@
+require 'rubygems'
+require 'trollop'
+require 'uri'
+
+require 'json/add/rails'
+require 'json/add/core'
+
+require File.join(File.dirname(__FILE__), 'couchpopulator.rb')
+require File.join(File.dirname(__FILE__), 'curl_adapter.rb')
+require File.join(File.dirname(__FILE__), 'generator.rb')
+require File.join(File.dirname(__FILE__), 'logger.rb')
+
+Dir.glob(File.join(File.dirname(__FILE__), 'couchpopulator/*.rb')).each {|f| require f }

data/lib/curl_adapter.rb

@@ -0,0 +1,36 @@
+class CurlAdapter
+  class Response
+    attr_reader :http_response_code
+    attr_reader :time_total
+
+    def initialize(http_response_code, time_total)
+      @http_response_code = http_response_code
+      @time_total = time_total
+    end
+
+    def inspect
+      "#{@http_response_code} #{@time_total} sec"
+    end
+  end
+
+  class Invoker
+    attr_reader :db_url
+
+    def initialize(db_url)
+      @db_url = db_url
+    end
+
+    def post(payload)
+      cmd = "curl -T - -X POST #{@db_url}/_bulk_docs -w\"%{http_code}\ %{time_total}\" -o out.file 2> /dev/null"
+      curl_io = IO.popen(cmd, "w+")
+      curl_io.puts payload
+      curl_io.close_write
+      result = CurlAdapter::Response.new(*curl_io.gets.split(" "))
+    end
+  end
+end
+
+# TODO:
+# Keep-Alive mit curl? wäre geil...
+
+

data/lib/generator.rb

@@ -0,0 +1,18 @@
+# Some nasty Array-Monkeypatches
+class Array
+  def rand
+    self[Kernel.rand(length)]
+  end unless Array.methods.include?("rand")
+  
+  # works like rand, but returns n random elements from self
+  # if n >= self.lenth, self is returned
+  def randn(n)
+    return self if n >= length
+    ret = []
+    while (ret.length < n) do
+      dummy = rand
+      ret << dummy unless ret.member?(dummy)
+    end
+    ret
+  end unless Array.methods.include?("randn")
+end

data/lib/logger.rb

@@ -0,0 +1,16 @@
+module CouchPopulator
+  class Logger
+    def initialize(logfile)
+      @out = logfile.empty? ? $stdout : File.new(logfile, "a")
+    end
+
+    def log(message)
+      t = Time.now
+      @out << "#{t.strftime("%Y-%m-%d %H:%M:%S")}:#{t.usec} :: #{message} \n"
+    end
+    
+    def <<(message)
+      log(message)
+    end
+  end  
+end

metadata

@@ -0,0 +1,91 @@
+--- !ruby/object:Gem::Specification 
+name: couchpopulator
+version: !ruby/object:Gem::Version 
+  version: 0.1.0
+platform: ruby
+authors: 
+- Sebastian Cohnen
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-11-16 00:00:00 +01:00
+default_executable: couchpopulator
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: json
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.2.0
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: trollop
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "1.15"
+    version: 
+description: flexible tool for populating CouchDB with generated documents
+email: sebastian.cohnen@gmx.net
+executables: 
+- couchpopulator
+extensions: []
+
+extra_rdoc_files: 
+- README.md
+files: 
+- bin/couchpopulator
+- executors/standard.rb
+- generators/example.rb
+- lib/couchpopulator.rb
+- lib/couchpopulator/base.rb
+- lib/couchpopulator/couch_helper.rb
+- lib/couchpopulator/initializer.rb
+- lib/couchpopulator/misc_helper.rb
+- lib/curl_adapter.rb
+- lib/generator.rb
+- lib/logger.rb
+- README.md
+has_rdoc: true
+homepage: http://github.com/tisba/couchpopulator
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+- lib
+- generators
+- executors
+- lib
+- generators
+- executors
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.5
+signing_key: 
+specification_version: 3
+summary: flexible tool for populating CouchDB with generated documents
+test_files: []
+