rubberband_flamethrower 0.2.1 → 0.2.2

data/README.rdoc

@@ -1,87 +1,51 @@
 = Rubberband Flamethrower
 
-Rubberband Flamethrower is a collection of scripts for dealing with faked Elastic Search data. 
-
-This readme is mostly out of date because I am changing this code into a gem but I'm leaving it here and will edit when the gem conversion is completed.
-
-The main script "inserter.rb" is a script for rapidly inserting test data into Elastic Search. It inserts fake "tweet" type objects into a "twitter" index on a local Elastic Search server at localhost:9200.  It runs in an infinite loop until you stop it. 
-
-The script "retriever.rb" also runs in a constant loop, doing a search on the tweets type in the twitter index for all objects within the date range between 2 and 3 minutes ago and reports the number of objects found.  This can be used to easily approximate the maximum speed obtainable for inserting to a local Elastic Search index for a given AWS box size.
+Rubberband Flamethrower is a gem for inserting faked data into an Elastic Search server.
 
 == Pre-Requisites
 
-This script has been written with ruby 1.9.3 in mind. It requires two gems: httparty and active_support. Httparty is used to send commands to the Elastic Search server.  Active Support is used for the JSON library in core_ext.
-
-== To run the inserter script:
-
-The script "inserter.rb" will create and store objects with a message, username, and postDate to a local Elastic Search index called "twitter" for the type "tweet". The message is composed of 6 to 16 random words and capped at 140 characters.
+=== Elastic Search
 
-	cd /path/to/repository/rubberband_flamethrower
-	ruby inserter.rb
+You should install and have an Elastic Search node running before trying to use this gem
 
-There are several word lists in the "words" folder which come from SCOWL http://wordlist.sourceforge.net/scowl-readme
+To download Elastic Search, unarchive, and start a node:
 
-The code uses the combination of the 10, 20, and 35 lists by default.
+	$ curl -k -L -o elasticsearch-0.20.6.tar.gz http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.6.tar.gz
+	$ tar -zxvf elasticsearch-0.20.6.tar.gz
+	$ ./elasticsearch-0.20.6/bin/elasticsearch -f
 
-== To run the retriever:
+=== Ruby
 
-The retriever script is set up to run in a continuous loop and report the number of objects inserted into Elastic Search during the specified time period. It defaults to a span of one minute (really one minute and one second) that occurred between two and three minutes ago.
+You do not need a rails project to use this gem though it is easier to use if you do.  It has been designed with ruby 1.9.1 and above in mind. The sample method of the Array class is used in the code and was not a part of the 1.8.7 release.
 
-	cd /path/to/repository/rubberband_flamethrower
-	ruby retriever.rb
+== Installation
 
-== Benchmarks
+install the gem manually
 
-Running one AWS Small Instance. I installed and started Elastic Search and then cloned this project onto the box and started the script running in a screen session.
+	gem install rubberband_flamethrower
 
-Running only the insert script during the time period being sampled by the retriever with it set to the default one minute (and one second) span.
+== Use
 
-A few samples from the run show that we are inserting anywhere from 6,000 to 13,000 documents in Elastic Search.
+Insert into Elastic Search by using the gem through irb
 
-	{"query":{"range":{"postDate":{"from":"20130401T20:50:24","to":"20130401T20:51:24"}}}}
-	11325
-	{"query":{"range":{"postDate":{"from":"20130401T20:50:25","to":"20130401T20:51:25"}}}}
-	11317
-	{"query":{"range":{"postDate":{"from":"20130401T21:08:01","to":"20130401T21:09:01"}}}}
-	6365
-	{"query":{"range":{"postDate":{"from":"20130401T21:08:02","to":"20130401T21:09:02"}}}}
-	6328
-	{"query":{"range":{"postDate":{"from":"20130401T21:12:33","to":"20130401T21:13:33"}}}}
-	12689
-	{"query":{"range":{"postDate":{"from":"20130401T21:12:34","to":"20130401T21:13:34"}}}}
-	12684
+	require 'rubberband_flamethrower'
+	RubberbandFlamethrower.send_batch
 
-Running the insert script and retriever script during the time period being sampled by the retriever for a short period did not seem to greatly affect the number, though running the retriever for a long duration did greatly degrade the performance of the inserts and increased load average on the box.
+The send_batch method can be configured by passing parameters. By default it will insert 5000 documents starting with document ID 1 into an Elastic Search index named "twitter" of type "tweet" into a server node located at http://localhost:9200.
 
-Running the retriever on a sample period of an hour (and one second) instead of one minute (and one second) we still see a fairly large variance in the number being inserted per hour and so far have only seen it increasing in time. This is a very small sample and too brief and investigation to draw any real conclusions yet.
+There are 5 parameters accepted by the send_batch method, all of which have a default value if left blank. Here are the parameters in order with their default values: (how_many=5000, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet")
 
-	{"query":{"range":{"postDate":{"from":"20130401T20:18:43","to":"20130401T21:18:43"}}}}
-	377978
-	{"query":{"range":{"postDate":{"from":"20130401T20:18:44","to":"20130401T21:18:44"}}}}
-	378122
-	{"query":{"range":{"postDate":{"from":"20130401T20:30:59","to":"20130401T21:30:59"}}}}
-	462394
-	{"query":{"range":{"postDate":{"from":"20130401T20:31:00","to":"20130401T21:31:00"}}}}
-	462503
-	{"query":{"range":{"postDate":{"from":"20130401T20:34:19","to":"20130401T21:34:19"}}}}
-	485449
-	{"query":{"range":{"postDate":{"from":"20130401T20:34:20","to":"20130401T21:34:20"}}}}
-	485577
-	{"query":{"range":{"postDate":{"from":"20130401T20:40:10","to":"20130401T21:40:10"}}}}
-	529912
-	{"query":{"range":{"postDate":{"from":"20130401T20:40:11","to":"20130401T21:40:11"}}}}
-	529981
+To Insert 10,000 instead of 5,000:
+	RubberbandFlamethrower.send_batch(10000)
 
+To Insert 5,000 starting with the ID 5001
+	RubberbandFlamethrower.send_batch(5000,5001)
 
-== Things to Look Into Or Do/Notes
- - Does message length varying account for some of the variance in insert speed? How much?
- - What role does the size of the pool of words used to construct the random sentences play in the time it takes to insert into the index?
- - What happens when we add more nodes to the Elastic Search cluster?
- - As the script runs longer and longer on the limited word set does it become increasingly faster or slower at indexing new content? To put it another way, does having a lot that already uses the same words make it faster or does it slow down because it now matches it up to that much more information.
- - How realistic is it to use Elastic Search nodes in the same manner as MySQL master/slave relationships where all selects are done from other nodes and all inserts on a master? Is this just common sense good practice, why didn't I see this mentioned anywhere?
- - These scripts need to be run from inside the rubberband_flamethrower directory because the script is using a relative link to the random words files.
- - Rescue error -> Errno::ECONNREFUSED -> in inserter.rb and report that the server is not running or not accepting connections, check your configuration
+To Insert 2,000 starting with the ID 1 to a server located at http://es.com:9200
+	RubberbandFlamethrower.send_batch(2000,1,"http://es.com:9200")
 
+To put your documents into an index named "facebook" instead of "twitter" with a type of "message" instead of "tweet"
+	RubberbandFlamethrower.send_batch(5000,1,"http://localhost:9200","facebook","message")
 
 == Contributing to rubberband_flamethrower
  

data/Rakefile

@@ -56,7 +56,7 @@ require File.dirname(__FILE__) + "/lib/rubberband_flamethrower"
 namespace :rubberband_flamethrower do
   desc "Insert fake data into Elastic Search, use arguments to change defaults"
   task :fire, :how_many, :starting_id, :server_url, :index, :type do |t, args|
-    args.with_defaults(:how_many => 1000, :starting_id => 1, :server_url => "http://localhost:9200", :index => "twitter", :type => "tweet")
+    args.with_defaults(:how_many => 5000, :starting_id => 1, :server_url => "http://localhost:9200", :index => "twitter", :type => "tweet")
     RubberbandFlamethrower.send_batch(args[:how_many].to_i, args[:starting_id].to_i, args[:server_url], args[:index], args[:type])
   end
 end

data/VERSION

@@ -1 +1 @@
-0.2.1
+0.2.2

data/lib/rubberband_flamethrower.rb

@@ -1,5 +1,5 @@
 module RubberbandFlamethrower
-  def self.send_batch(how_many, starting_id, server_url, index, type)
+  def self.send_batch(how_many=5000, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet")
     require "active_support/core_ext"
     require 'httparty'
     require File.dirname(__FILE__)+"/rubberband_flamethrower/data_generator.rb"

data/rubberband_flamethrower.gemspec

@@ -5,7 +5,7 @@
 
 Gem::Specification.new do |s|
   s.name = "rubberband_flamethrower"
-  s.version = "0.2.1"
+  s.version = "0.2.2"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Michael Orr"]

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rubberband_flamethrower
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.2
   prerelease: 
 platform: ruby
 authors:
@@ -13,7 +13,7 @@ date: 2013-04-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: httparty
-  requirement: &10372660 !ruby/object:Gem::Requirement
+  requirement: &21031020 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -21,10 +21,10 @@ dependencies:
         version: 0.10.2
   type: :runtime
   prerelease: false
-  version_requirements: *10372660
+  version_requirements: *21031020
 - !ruby/object:Gem::Dependency
   name: activesupport
-  requirement: &10372160 !ruby/object:Gem::Requirement
+  requirement: &21030520 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,10 +32,10 @@ dependencies:
         version: 3.2.13
   type: :runtime
   prerelease: false
-  version_requirements: *10372160
+  version_requirements: *21030520
 - !ruby/object:Gem::Dependency
   name: rdoc
-  requirement: &10371680 !ruby/object:Gem::Requirement
+  requirement: &21030040 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +43,10 @@ dependencies:
         version: '3.12'
   type: :development
   prerelease: false
-  version_requirements: *10371680
+  version_requirements: *21030040
 - !ruby/object:Gem::Dependency
   name: bundler
-  requirement: &10371140 !ruby/object:Gem::Requirement
+  requirement: &21029540 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -54,10 +54,10 @@ dependencies:
         version: 1.3.0
   type: :development
   prerelease: false
-  version_requirements: *10371140
+  version_requirements: *21029540
 - !ruby/object:Gem::Dependency
   name: jeweler
-  requirement: &10370600 !ruby/object:Gem::Requirement
+  requirement: &21029000 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -65,7 +65,7 @@ dependencies:
         version: 1.8.4
   type: :development
   prerelease: false
-  version_requirements: *10370600
+  version_requirements: *21029000
 description: Use to quickly fill up some indicies in Elastic Search and to retrieve
   statistics about insertion rates
 email: michael@cloudspace.com
@@ -115,7 +115,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 2499937689788223217
+      hash: 2029033906449151258
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements: