rubberband_flamethrower 0.1.0

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/Gemfile

@@ -0,0 +1,15 @@
+source "http://rubygems.org"
+# Add dependencies required to use your gem here.
+# Example:
+#   gem "activesupport", ">= 2.3.5"
+
+gem "httparty", "~> 0.10.2"
+gem "activesupport", "~> 3.2.13"
+
+# Add dependencies to develop your gem here.
+# Include everything needed to run rake, tests, features, etc.
+group :development do
+  gem "rdoc", "~> 3.12"
+  gem "bundler", "~> 1.3.0"
+  gem "jeweler", "~> 1.8.4"
+end

data/Gemfile.lock

@@ -0,0 +1,32 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    activesupport (3.2.13)
+      i18n (= 0.6.1)
+      multi_json (~> 1.0)
+    git (1.2.5)
+    httparty (0.10.2)
+      multi_json (~> 1.0)
+      multi_xml (>= 0.5.2)
+    i18n (0.6.1)
+    jeweler (1.8.4)
+      bundler (~> 1.0)
+      git (>= 1.2.5)
+      rake
+      rdoc
+    json (1.7.7)
+    multi_json (1.7.2)
+    multi_xml (0.5.3)
+    rake (10.0.4)
+    rdoc (3.12.2)
+      json (~> 1.4)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  activesupport (~> 3.2.13)
+  bundler (~> 1.3.0)
+  httparty (~> 0.10.2)
+  jeweler (~> 1.8.4)
+  rdoc (~> 3.12)

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2013 Michael Orr
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,100 @@
+= Rubberband Flamethrower
+
+Rubberband Flamethrower is a collection of scripts for dealing with faked Elastic Search data. 
+
+This readme is mostly out of date because I am changing this code into a gem but I'm leaving it here and will edit when the gem conversion is completed.
+
+The main script "inserter.rb" is a script for rapidly inserting test data into Elastic Search. It inserts fake "tweet" type objects into a "twitter" index on a local Elastic Search server at localhost:9200.  It runs in an infinite loop until you stop it. 
+
+The script "retriever.rb" also runs in a constant loop, doing a search on the tweets type in the twitter index for all objects within the date range between 2 and 3 minutes ago and reports the number of objects found.  This can be used to easily approximate the maximum speed obtainable for inserting to a local Elastic Search index for a given AWS box size.
+
+== Pre-Requisites
+
+This script has been written with ruby 1.9.3 in mind. It requires two gems: httparty and active_support. Httparty is used to send commands to the Elastic Search server.  Active Support is used for the JSON library in core_ext.
+
+== To run the inserter script:
+
+The script "inserter.rb" will create and store objects with a message, username, and postDate to a local Elastic Search index called "twitter" for the type "tweet". The message is composed of 6 to 16 random words and capped at 140 characters.
+
+	cd /path/to/repository/rubberband_flamethrower
+	ruby inserter.rb
+
+There are several word lists in the "words" folder which come from SCOWL http://wordlist.sourceforge.net/scowl-readme
+
+The code uses the combination of the 10, 20, and 35 lists by default.
+
+== To run the retriever:
+
+The retriever script is set up to run in a continuous loop and report the number of objects inserted into Elastic Search during the specified time period. It defaults to a span of one minute (really one minute and one second) that occurred between two and three minutes ago.
+
+	cd /path/to/repository/rubberband_flamethrower
+	ruby retriever.rb
+
+== Benchmarks
+
+Running one AWS Small Instance. I installed and started Elastic Search and then cloned this project onto the box and started the script running in a screen session.
+
+Running only the insert script during the time period being sampled by the retriever with it set to the default one minute (and one second) span.
+
+A few samples from the run show that we are inserting anywhere from 6,000 to 13,000 documents in Elastic Search.
+
+	{"query":{"range":{"postDate":{"from":"20130401T20:50:24","to":"20130401T20:51:24"}}}}
+	11325
+	{"query":{"range":{"postDate":{"from":"20130401T20:50:25","to":"20130401T20:51:25"}}}}
+	11317
+	{"query":{"range":{"postDate":{"from":"20130401T21:08:01","to":"20130401T21:09:01"}}}}
+	6365
+	{"query":{"range":{"postDate":{"from":"20130401T21:08:02","to":"20130401T21:09:02"}}}}
+	6328
+	{"query":{"range":{"postDate":{"from":"20130401T21:12:33","to":"20130401T21:13:33"}}}}
+	12689
+	{"query":{"range":{"postDate":{"from":"20130401T21:12:34","to":"20130401T21:13:34"}}}}
+	12684
+
+Running the insert script and retriever script during the time period being sampled by the retriever for a short period did not seem to greatly affect the number, though running the retriever for a long duration did greatly degrade the performance of the inserts and increased load average on the box.
+
+Running the retriever on a sample period of an hour (and one second) instead of one minute (and one second) we still see a fairly large variance in the number being inserted per hour and so far have only seen it increasing in time. This is a very small sample and too brief and investigation to draw any real conclusions yet.
+
+	{"query":{"range":{"postDate":{"from":"20130401T20:18:43","to":"20130401T21:18:43"}}}}
+	377978
+	{"query":{"range":{"postDate":{"from":"20130401T20:18:44","to":"20130401T21:18:44"}}}}
+	378122
+	{"query":{"range":{"postDate":{"from":"20130401T20:30:59","to":"20130401T21:30:59"}}}}
+	462394
+	{"query":{"range":{"postDate":{"from":"20130401T20:31:00","to":"20130401T21:31:00"}}}}
+	462503
+	{"query":{"range":{"postDate":{"from":"20130401T20:34:19","to":"20130401T21:34:19"}}}}
+	485449
+	{"query":{"range":{"postDate":{"from":"20130401T20:34:20","to":"20130401T21:34:20"}}}}
+	485577
+	{"query":{"range":{"postDate":{"from":"20130401T20:40:10","to":"20130401T21:40:10"}}}}
+	529912
+	{"query":{"range":{"postDate":{"from":"20130401T20:40:11","to":"20130401T21:40:11"}}}}
+	529981
+
+
+== Things to Look Into Or Do/Notes
+ - Does message length varying account for some of the variance in insert speed? How much?
+ - What role does the size of the pool of words used to construct the random sentences play in the time it takes to insert into the index?
+ - What happens when we add more nodes to the Elastic Search cluster?
+ - As the script runs longer and longer on the limited word set does it become increasingly faster or slower at indexing new content? To put it another way, does having a lot that already uses the same words make it faster or does it slow down because it now matches it up to that much more information.
+ - How realistic is it to use Elastic Search nodes in the same manner as MySQL master/slave relationships where all selects are done from other nodes and all inserts on a master? Is this just common sense good practice, why didn't I see this mentioned anywhere?
+ - These scripts need to be run from inside the rubberband_flamethrower directory because the script is using a relative link to the random words files.
+ - Rescue error -> Errno::ECONNREFUSED -> in inserter.rb and report that the server is not running or not accepting connections, check your configuration
+
+
+== Contributing to rubberband_flamethrower
+ 
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
+* Fork the project.
+* Start a feature/bugfix branch.
+* Commit and push until you are happy with your contribution.
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+== Copyright
+
+Copyright (c) 2013 Michael Orr. See LICENSE.txt for
+further details.
+

data/Rakefile

@@ -0,0 +1,61 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "rubberband_flamethrower"
+  gem.homepage = "http://github.com/imightbeinatree/rubberband_flamethrower"
+  gem.license = "MIT"
+  gem.summary = %Q{Rapidly Insert Fake Data into Elastic Search}
+  gem.description = %Q{Use to quickly fill up some indicies in Elastic Search and to retrieve statistics about insertion rates}
+  gem.email = "michael@cloudspace.com"
+  gem.authors = ["Michael Orr"]
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+#require 'rcov/rcovtask'
+#Rcov::RcovTask.new do |test|
+#  test.libs << 'test'
+#  test.pattern = 'test/**/test_*.rb'
+#  test.verbose = true
+#  test.rcov_opts << '--exclude "gems/*"'
+#end
+
+task :default => :test
+
+require 'rdoc/task'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "rubberband_flamethrower #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+require File.dirname(__FILE__) + "/lib/rubberband_flamethrower"
+namespace :rubberband_flamethrower do
+  desc "Insert fake data into Elastic Search"
+  task :fire do
+    RubberbandFlamethrower.start_insert
+  end
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/rubberband_flamethrower/data_generator.rb

@@ -0,0 +1,63 @@
+# This class is specially designed to generate random data to be inserted into Elastic Search
+# It creates a JSON object that approaxiamates data like you would fine in a tweet
+# with fields for a message (max 140 characters), username, and post_date.
+# The post_date format is parsable as a date object by default by Elastic Search
+module RubberbandFlamethrower
+  class DataGenerator
+    attr_accessor :word_list
+  
+    # the WORD_FILES constant is an array of included word files
+    # which will be used to create the pool of random words used for data generation.
+    # You can uncomment or comment particular files to change the size of the pool of words.
+    # Please see the README file in the words folder for more information about the lists.
+    WORD_FILES = [
+      # "/words/american-words.95",
+      # "/words/american-words.80",
+      # "/words/american-words.70",
+      # "/words/american-words.60",
+      # "/words/american-words.55",
+      # "/words/american-words.50",
+      # "/words/american-words.40",
+      "/words/american-words.35",
+      "/words/american-words.20",
+      "/words/american-words.10"
+      ]
+
+    # Will initialize the word_list variable with an array of all the words contained in the WORD_FILES array   
+    def initialize
+      self.word_list = []
+      WORD_FILES.each do |word_file|
+        contents = File.read(File.dirname(__FILE__)+word_file)
+        self.word_list = word_list + contents.split(/\n/)
+      end
+    end
+  
+    # create a message from between 6 and 16 random words that maxes at 140 characters and ends with a period
+    # @return [String]
+    def random_tweet
+      number_of_words = 6 + rand(10)
+      ((number_of_words.times.map{word_list.sample}.join(" "))[0,139])+"."
+    end
+  
+    # create a random value to be used as a username
+    # the return value is one random word, only letters and numbers allowed
+    # @return [String]
+    def random_username
+      word_list.sample.gsub(/[^0-9a-z]/i, '')
+    end
+  
+    # create an Elastic Search friendly timestamp for right now
+    # @return [String]
+    def current_timestamp
+      Time.now.strftime "%Y%m%dT%H:%M:%S"
+    end  
+  
+    # generate a JSON object that contains a message, username, and post_date
+    # intended to be passed as insert data to an elastic search server
+    # @return [JSON]
+    def generate_random_insert_data
+      {message: "#{random_tweet}", username: "#{random_username}", post_date: "#{current_timestamp}"}.to_json
+    end
+  
+  end
+end

data/lib/rubberband_flamethrower/inserter.rb

@@ -0,0 +1,37 @@
+# Rubberband Flamethrower is a collection of scripts for dealing with faked Elastic Search data. 
+# This file is the main script for inserting fake data 
+# It inserts fake "tweet" type objects into a "twitter" index 
+# on a local Elastic Search server at localhost:9200.  
+# It runs in an infinite loop until you stop it.
+#
+#
+# Author::    Michael Orr 
+#             email - michael@cloudspace.com 
+#             twitter - @imbiat
+# Copyright:: Copyright (c) 2013
+# License::   MIT License
+
+
+require_relative "data_generator.rb"
+
+# a unique ID must be provided for each document stored in Elastic Search
+# since we are looping, we set a starting id and it will be incremented in the loop
+id = 2
+
+# initialize the random data generator object
+data = DataGenerator.new
+
+# start infinite loop
+while true do  
+  
+  # generate a piece of random data to insert that approxiamates a tweet
+  insert_data = data.generate_random_insert_data
+  puts insert_data
+  
+  # insert the random data into local elastic search index "twitter" as type "tweet" with set id
+  response = HTTParty.put("http://localhost:9200/twitter/tweet/#{id}", body: insert_data) 
+  puts response.body
+  
+  #increment the insert id
+  id = id + 1
+end

data/lib/rubberband_flamethrower/retriever.rb

@@ -0,0 +1,37 @@
+# Rubberband Flamethrower is a collection of scripts for dealing with faked Elastic Search data. 
+# This file is the main script for retrieving fake data 
+# When running it will loop continuously
+# doing a search on the tweets type in the twitter index
+# for all objects within the date range between 2 and 3 minutes ago
+# and reports the number of objects found.
+#
+# This can be used to easily approximate the maximum speed obtainable 
+# for inserting to a local Elastic Search index for a given AWS box size.
+#
+#
+# Author::    Michael Orr 
+#             email - michael@cloudspace.com 
+#             twitter - @imbiat
+# Copyright:: Copyright (c) 2013
+# License::   MIT License
+
+
+require 'httparty'
+require "active_support/core_ext"
+
+# start infinite loop
+while true do
+
+  # set up start and end range values for query
+  range_start = (Time.now - 3.minutes).strftime "%Y%m%dT%H:%M:%S"
+  range_end = (Time.now - 2.minutes).strftime "%Y%m%dT%H:%M:%S"
+
+  # set up json for elastic search range query
+  query = {query: {range: { postDate: {from: "#{range_start}", to: "#{range_end}"}}}}.to_json
+  puts query
+
+  # perform elastic search query and report results
+  response = HTTParty.get("http://localhost:9200/twitter/tweet/_search", body: query)
+  parsed = JSON.parse(response.body)
+  puts "hits found: #{parsed["hits"]["total"]}"
+end

data/lib/rubberband_flamethrower/words/README

@@ -0,0 +1,48 @@
+These word lists are from the SCOWL wordlist set:
+
+http://downloads.sourceforge.net/wordlist/scowl-7.1.tar.gz
+http://wordlist.sourceforge.net/scowl-readme
+http://wordlist.sourceforge.net/scowl-7.1-readme
+
+The SCOWL word lists are used primarily for spellchecking.
+
+I use just the 10, 20, and 35 lists by default.
+
+The following information appears in the scowl-readme file linked above.
+
+And here is a count on the number of in each spelling category
+(american + english spelling category):
+
+  Size   Words       Names    Running Total  %
+   10    4,427          15        4,442     0.7
+   20    8,122           0       12,564     1.9
+   35   37,251         224       50,039     7.7
+   40    6,802         503       57,344     8.8
+   50   24,505      15,455       97,304    14.9
+   55    6,555           0      103,859    15.9
+   60   13,633         775      118,267    18.1
+   70   35,507       7,747      161,521    24.8
+   80  143,791      33,293      338,605    51.9
+   95  227,056      86,814      652,475   100.0
+
+COPYRIGHT, SOURCES, and CREDITS:
+
+The collective work is Copyright 2000-2011 by Kevin Atkinson as well
+as any of the copyrights mentioned below:
+
+  Copyright 2000-2011 by Kevin Atkinson
+
+  Permission to use, copy, modify, distribute and sell these word
+  lists, the associated scripts, the output created from the scripts,
+  and its documentation for any purpose is hereby granted without fee,
+  provided that the above copyright notice appears in all copies and
+  that both that copyright notice and this permission notice appear in
+  supporting documentation. Kevin Atkinson makes no representations
+  about the suitability of this array for any purpose. It is provided
+  "as is" without express or implied warranty.
+
+Alan Beale <biljir@pobox.com> also deserves special credit as he has,
+in addition to providing the 12Dicts package and being a major
+contributor to the ENABLE word list, given me an incredible amount of
+feedback and created a number of special lists (those found in the
+Supplement) in order to help improve the overall quality of SCOWL.

data/lib/rubberband_flamethrower/words/american-words.10

@@ -0,0 +1,35 @@
+afterward
+among
+analog
+apologize
+behavior
+catalog
+center
+color
+color's
+colors
+defense
+favor
+favorite
+flavor
+gray
+judgment
+labeled
+labeling
+labor
+learned
+organization
+organize
+organized
+organizes
+organizing
+realize
+realized
+realizes
+realizing
+recognize
+recognized
+recognizes
+recognizing
+rumor
+spelled

data/lib/rubberband_flamethrower/words/american-words.20

@@ -0,0 +1,169 @@
+adapter
+aging
+analyze
+analyzed
+analyzes
+analyzing
+apologized
+apologizes
+apologizing
+appall
+appalls
+ass
+authorization
+authorize
+authorized
+authorizes
+authorizing
+ax
+ax's
+buses
+canceled
+canceling
+catalog's
+catalogs
+center's
+centered
+centering
+centers
+civilization
+civilize
+civilized
+civilizes
+civilizing
+colored
+coloring
+computerize
+computerized
+computerizes
+computerizing
+counseled
+counseling
+cozy
+criticize
+criticized
+criticizes
+criticizing
+dialed
+dialing
+dialings
+dreamed
+emphasize
+emphasized
+emphasizes
+emphasizing
+endeavor
+favor's
+favorable
+favored
+favoring
+favorite's
+favorites
+favors
+fiber
+fiber's
+fibers
+finalize
+finalized
+finalizes
+finalizing
+fulfill
+fulfills
+generalization
+generalization's
+generalizations
+generalize
+generalized
+generalizes
+generalizing
+honor
+honor's
+honored
+honoring
+honors
+humor
+initialization
+initialize
+initialized
+initializes
+initializing
+inquire
+inquired
+inquires
+inquiries
+inquiring
+inquiry
+inquiry's
+judgment's
+judgments
+license's
+liter
+marvelous
+maximize
+minimize
+modeled
+modeling
+modelings
+mold
+neighbor
+neighbor's
+neighborhood
+neighbors
+offense
+offense's
+offenses
+optimization
+optimize
+optimized
+optimizes
+optimizing
+organization's
+organizations
+organizer
+organizer's
+organizers
+patronize
+patronized
+patronizes
+patronizing
+practiced
+practicing
+privatization
+publicize
+publicized
+publicizes
+publicizing
+realization
+recognizable
+rumor's
+rumored
+rumoring
+rumors
+signaled
+signaling
+skeptical
+specialize
+specialized
+specializes
+specializing
+specialty
+spoiled
+standardize
+standardized
+standardizes
+standardizing
+subsidize
+subsidized
+subsidizes
+subsidizing
+summarize
+summarized
+summarizes
+summarizing
+sympathize
+theater
+traveled
+traveling
+travelings
+unauthorized
+whiskey