rubberband_flamethrower 0.4.0 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (7) hide show
  1. data/README.md +30 -7
  2. data/VERSION +1 -1
  3. data/bin/flamethrower +4 -4
  4. data/lib/rubberband_flamethrower/flamethrower.rb +23 -3
  5. data/lib/rubberband_flamethrower.rb +57 -13
  6. data/rubberband_flamethrower.gemspec +1 -1
  7. metadata +12 -12
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Rubberband Flamethrower
2
2
 
3
- Rubberband Flamethrower is a gem for inserting faked data into an Elastic Search server and providing basic benchmarks. It creates and inserts fake data objects with three fields (message, username, and post_date) and times the process. It inserts in batches of 5,000 objects by default but can be configured to insert any number of objects. Here is an sample generated data object:
3
+ Rubberband Flamethrower is a gem for inserting faked data into an Elastic Search server and providing basic benchmarks. It creates and inserts fake data objects with three fields (message, username, and post_date) and times the process. It inserts in batches of 500 objects by default but can be configured to insert any number of objects. Here is an sample generated data object:
4
4
 
5
5
  {
6
6
  "message":"utilizing plowed popularizing demeanor anesthesia specializes chaperon pedaling.",
@@ -29,7 +29,7 @@ Start an Elastic Search node:
29
29
 
30
30
  ### Ruby
31
31
 
32
- You do not need a rails project to use this gem though it is easier to use if you do. It has been designed with ruby 1.9.1 and above in mind. The sample method of the Array class is used in the code and was not a part of the 1.8.7 release.
32
+ It has been designed with ruby 1.9.1 and above in mind. The sample method of the Array class is used in the code and was not a part of the 1.8.7 release.
33
33
 
34
34
  ## Installation
35
35
 
@@ -43,9 +43,9 @@ Or if you will be using this as a part of a Rails project, you can add the gem t
43
43
 
44
44
  ### Command Line Executable
45
45
 
46
- Once the gem is installed and you have an Elastic Search server running you are ready to begin inserting fake data. You can run the gem from the command line using the "flamethrower" command. The command without any arguments will display the help screen:
46
+ Once the gem is installed and you have an Elastic Search server running you are ready to begin inserting fake data. You can run the gem from the command line using the "flamethrower" command.
47
47
 
48
- flamethrower
48
+ #### Fire
49
49
 
50
50
  To start a batch insert into the local Elastic Search server you add the argument "fire" to the command:
51
51
 
@@ -53,9 +53,9 @@ To start a batch insert into the local Elastic Search server you add the argumen
53
53
 
54
54
  By default it will insert 5000 documents starting with document ID 1 into an Elastic Search index named "twitter" of type "tweet" into a server node located at http://localhost:9200.
55
55
 
56
- You can configure what is inserted by passing additional parameters. There are 5 parameters accepted by the `flamethrower fire` command, all of which have a default value if left blank. Here are the parameters in order with their default values: (how_many=5000, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet")
56
+ You can configure what is inserted by passing additional parameters. The parameters accepted by the `flamethrower fire` command all have a default value if left blank. Here are the parameters in order with their default values: (how_many=500, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet")
57
57
 
58
- To Insert 10,000 instead of 5,000:
58
+ To Insert 10,000 instead of 500:
59
59
 
60
60
  flamethrower fire 10000
61
61
 
@@ -69,8 +69,31 @@ To Insert 2,000 starting with the ID 1 to a server located at http://es.com:9200
69
69
 
70
70
  To put your documents into an index named "facebook" instead of "twitter" with a type of "message" instead of "tweet"
71
71
 
72
- flamethrower fire 5000 1 "http://localhost:9200" "facebook" "message"
72
+ flamethrower fire 500 1 "http://localhost:9200" "facebook" "message"
73
+
74
+ #### Auto
75
+
76
+ The "auto" argument can be used to repeatedly run the "flamethrower fire" command, timing each run. By default it will run the command 3 times.
77
+
78
+ flamethrower auto
79
+
80
+ This can be configured much like the above example, though there is one additional parameter, which is supplied first and represents the number of times to run the "flamethrower fire" command. Here are the parameters in order with their default values: (how_many_batches=3, per_batch=5000, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet", id_overwrite="n")
81
+
82
+ To run the "flamethrower fire" command 5 times in a row instead of the default 3:
83
+
84
+ flamethrower auto 5
85
+
86
+ To run the "flamethrower fire" command 5 times, inserting 5,000 objects each time:
87
+
88
+ flamethrower auto 5 5000
89
+
90
+ The id_overwrite parameter determines the ID strategy used for subsequent batches in the auto command. When set to \"n\" (which it is by default) each batch will be writing new data with fresh IDs to the Elastic Search server, simulating a system where data is constantly being inserted and not updated. 5 batches of 500 with an \"n\" would use the IDs 1-2,500. When it is set to \"y\" each batch will simulate overwriting existing data in the Elastic Search server, simulating a system where data is constantly being updated (after the initial batch). 5 batches of 500 with a setting of \"y\" would use the IDs 1-500 on each batch.
91
+
92
+ #### Help
93
+
94
+ The command with the argument "help" or without any arguments will display the help screen:
73
95
 
96
+ flamethrower help
74
97
 
75
98
  ### IRB/Ruby Scripts
76
99
 
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.4.0
1
+ 0.4.1
data/bin/flamethrower CHANGED
@@ -7,15 +7,15 @@ require 'rubberband_flamethrower'
7
7
 
8
8
  args = ARGV.dup
9
9
  ARGV.clear
10
- command = args.shift.strip rescue 'help'
10
+ command = args.shift.strip rescue 'auto'
11
11
 
12
12
  case command
13
13
  when "fire"
14
14
  RubberbandFlamethrower.fire(*args)
15
- when "auto"
16
- RubberbandFlamethrower.auto(*args)
17
- else
15
+ when "help"
18
16
  RubberbandFlamethrower.help
17
+ else
18
+ RubberbandFlamethrower.auto(*args)
19
19
  end
20
20
 
21
21
  #binary adapted from old heroku source
data/lib/rubberband_flamethrower/flamethrower.rb CHANGED
@@ -1,9 +1,30 @@
1
1
  # This class is designed to insert objects created with the DataGenerator class into Elastic Search
2
+
3
+ require "active_support/core_ext"
4
+ require 'httparty'
5
+ require "benchmark"
6
+ require File.dirname(__FILE__)+"/data_generator.rb"
7
+
2
8
  module RubberbandFlamethrower
3
9
  class Flamethrower
10
+
11
+ # Batch inserts randomly generated JSON data objects into an Elastic Search server.
12
+ # Example of the object inserted:
13
+ # {
14
+ # "message":"utilizing plowed popularizing demeanor anesthesia specializes chaperon pedaling.",
15
+ # "username":"pummeling",
16
+ # "post_date":"20130408T15:41:28"
17
+ # }
18
+ #
19
+ # @param [Integer] how_many - how many randomly generated data objects to insert
20
+ # @param [Integer] starting_id - starting id for randomly generated data objects, will increment from this number
21
+ # @param [String] server_url - url of the Elastic Search server
22
+ # @param [String] index - name of the Elastic Search index to insert data into
23
+ # @param [String] type - name of the Elastic Search type to insert data into
24
+ # @param [Integer] printing_level - 0 = no printing | 1 = progress dots | 2 = debug level
4
25
  def fire(how_many, starting_id, server_url, index, type, printing_level=0)
5
26
 
6
-
27
+ # this will constantly flush the printing to the display so the progress dots appear for each insert
7
28
  ($stdout.sync = true) if printing_level == 1
8
29
 
9
30
  # a unique ID must be provided for each document stored in Elastic Search
@@ -12,8 +33,8 @@ module RubberbandFlamethrower
12
33
  # initialize the random data generator object
13
34
  data = DataGenerator.new
14
35
 
36
+ # loop through inserting random data
15
37
  how_many.to_i.times do |i|
16
-
17
38
  # generate a piece of random data to insert that approxiamates a tweet
18
39
  insert_data = data.generate_random_insert_data
19
40
  (puts insert_data) if printing_level == 2
@@ -27,6 +48,5 @@ module RubberbandFlamethrower
27
48
  id = id + 1
28
49
  end
29
50
  end
30
-
31
51
  end
32
52
  end
data/lib/rubberband_flamethrower.rb CHANGED
@@ -1,9 +1,12 @@
1
1
  module RubberbandFlamethrower
2
- def self.fire(how_many=5000, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet")
3
- require "active_support/core_ext"
4
- require 'httparty'
5
- require "benchmark"
6
- require File.dirname(__FILE__)+"/rubberband_flamethrower/data_generator.rb"
2
+
3
+ # Benchmarks a call to the fire method (which inserts a batch of random data into Elastic Search)
4
+ # @param [Integer] how_many - how many randomly generated data objects to insert
5
+ # @param [Integer] starting_id - starting id for randomly generated data objects, will increment from this number
6
+ # @param [String] server_url - url of the Elastic Search server
7
+ # @param [String] index - name of the Elastic Search index to insert data into
8
+ # @param [String] type - name of the Elastic Search type to insert data into
9
+ def self.fire(how_many=500, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet")
7
10
  require File.dirname(__FILE__)+"/rubberband_flamethrower/flamethrower.rb"
8
11
  flamethrower = Flamethrower.new
9
12
  time = Benchmark.measure do
@@ -14,23 +17,64 @@ module RubberbandFlamethrower
14
17
  puts time
15
18
  end
16
19
 
17
- def self.auto(how_many_batches=3, per_batch=5000, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet")
18
- require "active_support/core_ext"
19
- require 'httparty'
20
- require "benchmark"
21
- require File.dirname(__FILE__)+"/rubberband_flamethrower/data_generator.rb"
20
+ # Benchmarks a series of calls to the fire method (which inserts a batch of random data into Elastic Search)
21
+ # @param [Integer] how_many_batches - how many batches to run
22
+ # @param [Integer] per_batch - how many randomly generated data objects to insert
23
+ # @param [Integer] starting_id - starting id for randomly generated data objects, will increment from this number
24
+ # @param [String] server_url - url of the Elastic Search server
25
+ # @param [String] index - name of the Elastic Search index to insert data into
26
+ # @param [String] type - name of the Elastic Search type to insert data into
27
+ def self.auto(how_many_batches=3, per_batch=500, starting_id=1, server_url="http://localhost:9200", index="twitter", type="tweet", id_overwrite="n")
22
28
  require File.dirname(__FILE__)+"/rubberband_flamethrower/flamethrower.rb"
23
29
  flamethrower = Flamethrower.new
30
+ puts "#{per_batch} documents inserted into Elastic Search per set"
24
31
  Benchmark.bm(8) do |x|
25
32
  how_many_batches.to_i.times do |i|
26
- x.report("set #{i+1}:") { flamethrower.fire(per_batch, starting_id, server_url, index, type) }
27
- starting_id = starting_id.to_i + per_batch.to_i
33
+ x.report("set #{i+1} of #{how_many_batches}:") { flamethrower.fire(per_batch, starting_id, server_url, index, type) }
34
+ # increment the starting id by the batch size on each loop unless "id_overwrite" is set to "y"
35
+ # when it is set to "n" (which it is by default)
36
+ # each batch will be writing new data with fresh IDs to the Elastic Search server
37
+ # simulating a system where data is constantly being inserted and not updated
38
+ # when it is set to "y" each batch will simulate overwriting existing data in the Elastic Search server
39
+ # simulating a system where data is constantly being updated (after the initial batch)
40
+ (starting_id = starting_id.to_i + per_batch.to_i) unless id_overwrite == "y"
28
41
  end
29
42
  end
30
43
  end
31
44
 
45
+ # Displays help menu of the available help menu commands
32
46
  def self.help
33
- puts "Flamethrower Commands Available:\n\nflamethrower #display this help message\nflamethrower fire #sends a batch to the elastic search server"
47
+ puts "Rubberband Flamethrower is a gem for inserting faked data into an Elastic Search server"
48
+ puts "and providing basic benchmarks. It creates and inserts fake data objects with three"
49
+ puts "fields (message, username, and post_date) and reports timing information."
50
+
51
+ puts "\n\nFlamethrower Commands Available:\n\n"
52
+ puts "flamethrower fire #benchmark a batch insert of data to an elastic search server"
53
+ puts "flamethrower auto #benchmark a series of batch inserts to an elastic search server"
54
+ puts "flamethrower help #display this help message"
55
+
56
+ puts "\n\nThe fire and auto commands can be configured by passing arguments."
57
+ puts "The parameters accepted by fire and auto all have a default value if left blank."
58
+
59
+ puts "\n\n\"fire\" parameters in order with their default values:"
60
+ puts "(how_many=500, starting_id=1, server_url=http://localhost:9200, index=twitter, type=tweet)"
61
+
62
+ puts "\n\n\"auto\" parameters in order with their default values:"
63
+ puts "(how_many_batches=3, per_batch=500, starting_id=1, server_url=http://localhost:9200, index=twitter, type=tweet, id_overwrite=n)"
64
+
65
+ puts "\n\nUsage Examples With Parameters:"
66
+ puts "flamethrower fire 10000 #To Insert 10,000 instead of 500"
67
+ puts "flamethrower fire 5000 5001 #To Insert 5,000 starting with the ID 5001"
68
+ puts "flamethrower fire 2000 1 \"http://es.com:9200\" #Elastic Search Node located at http://es.com:9200"
69
+ puts "flamethrower fire 500 1 \"http://localhost:9200\" \"facebook\" \"message\""
70
+ puts "\t#Insert into an index named \"facebook\" instead of \"twitter\""
71
+ puts "\t#with a type of \"message\" instead of \"tweet\""
72
+
73
+ puts "\n\nThe id_overwrite parameter determines the ID strategy used for subsequent batches in the auto command."
74
+ puts "When set to \"n\" (which it is by default)each batch will be writing new data with fresh IDs to the Elastic"
75
+ puts "Search server, simulating a system where data is constantly being inserted and not updated. When it is set"
76
+ puts "to \"y\" each batch will simulate overwriting existing data in the Elastic Search server, simulating a"
77
+ puts "system where data is constantly being updated (after the initial batch)."
34
78
  end
35
79
 
36
80
  end
data/rubberband_flamethrower.gemspec CHANGED
@@ -5,7 +5,7 @@
5
5
 
6
6
  Gem::Specification.new do |s|
7
7
  s.name = "rubberband_flamethrower"
8
- s.version = "0.4.0"
8
+ s.version = "0.4.1"
9
9
 
10
10
  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
11
11
  s.authors = ["Michael Orr"]
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rubberband_flamethrower
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.0
4
+ version: 0.4.1
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -13,7 +13,7 @@ date: 2013-04-09 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: httparty
16
- requirement: &17477820 !ruby/object:Gem::Requirement
16
+ requirement: &15221100 !ruby/object:Gem::Requirement
17
17
  none: false
18
18
  requirements:
19
19
  - - ~>
@@ -21,10 +21,10 @@ dependencies:
21
21
  version: 0.10.2
22
22
  type: :runtime
23
23
  prerelease: false
24
- version_requirements: *17477820
24
+ version_requirements: *15221100
25
25
  - !ruby/object:Gem::Dependency
26
26
  name: activesupport
27
- requirement: &17477280 !ruby/object:Gem::Requirement
27
+ requirement: &15219640 !ruby/object:Gem::Requirement
28
28
  none: false
29
29
  requirements:
30
30
  - - ~>
@@ -32,10 +32,10 @@ dependencies:
32
32
  version: 3.2.13
33
33
  type: :runtime
34
34
  prerelease: false
35
- version_requirements: *17477280
35
+ version_requirements: *15219640
36
36
  - !ruby/object:Gem::Dependency
37
37
  name: rdoc
38
- requirement: &17476800 !ruby/object:Gem::Requirement
38
+ requirement: &15217040 !ruby/object:Gem::Requirement
39
39
  none: false
40
40
  requirements:
41
41
  - - ~>
@@ -43,10 +43,10 @@ dependencies:
43
43
  version: '3.12'
44
44
  type: :development
45
45
  prerelease: false
46
- version_requirements: *17476800
46
+ version_requirements: *15217040
47
47
  - !ruby/object:Gem::Dependency
48
48
  name: bundler
49
- requirement: &17476280 !ruby/object:Gem::Requirement
49
+ requirement: &15225080 !ruby/object:Gem::Requirement
50
50
  none: false
51
51
  requirements:
52
52
  - - ~>
@@ -54,10 +54,10 @@ dependencies:
54
54
  version: 1.3.0
55
55
  type: :development
56
56
  prerelease: false
57
- version_requirements: *17476280
57
+ version_requirements: *15225080
58
58
  - !ruby/object:Gem::Dependency
59
59
  name: jeweler
60
- requirement: &17475800 !ruby/object:Gem::Requirement
60
+ requirement: &15239460 !ruby/object:Gem::Requirement
61
61
  none: false
62
62
  requirements:
63
63
  - - ~>
@@ -65,7 +65,7 @@ dependencies:
65
65
  version: 1.8.4
66
66
  type: :development
67
67
  prerelease: false
68
- version_requirements: *17475800
68
+ version_requirements: *15239460
69
69
  description: Use to quickly fill up some indicies in Elastic Search and to retrieve
70
70
  statistics about insertion rates
71
71
  email: michael@cloudspace.com
@@ -117,7 +117,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
117
117
  version: '0'
118
118
  segments:
119
119
  - 0
120
- hash: 3868063792966642724
120
+ hash: 1869112594906546010
121
121
  required_rubygems_version: !ruby/object:Gem::Requirement
122
122
  none: false
123
123
  requirements: