dynamo-autoscale 0.1.4 → 0.2

data/.gitignore

@@ -1,4 +1,4 @@
-config/aws.yml
+config/dynamo-autoscale.yml
 aws.yml
 data/
 pkg/

data/CHANGELOG

@@ -1,3 +1,9 @@
+v0.2
+
+- Completely reworked configuration. Got rid of the massive number of flags and
+	options required to launch dynamo-autoscale and moved them into a config file.
+- Added the ability to send and email whenever a scale happens.
+
 v0.1.4
 
 - Fixed a bug that stopped all rule evaluation after the first rule passed. Now

data/Gemfile.lock

@@ -1,19 +1,25 @@
 PATH
   remote: .
   specs:
-    dynamo-autoscale (0.1)
+    dynamo-autoscale (0.1.4)
+      activesupport
       aws-sdk
       colored
+      pony
       rbtree
       ruby-prof
 
 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (3.2.13)
-      i18n (= 0.6.1)
-      multi_json (~> 1.0)
-    aws-sdk (1.11.2)
+    activesupport (4.0.0)
+      i18n (~> 0.6, >= 0.6.4)
+      minitest (~> 4.2)
+      multi_json (~> 1.3)
+      thread_safe (~> 0.1)
+      tzinfo (~> 0.3.37)
+    atomic (1.1.10)
+    aws-sdk (1.11.3)
       json (~> 1.4)
       nokogiri (< 1.6.0)
       uuidtools (~> 2.1)
@@ -21,11 +27,19 @@ GEM
     coderay (1.0.9)
     colored (1.2)
     diff-lcs (1.2.4)
-    i18n (0.6.1)
+    i18n (0.6.4)
     json (1.8.0)
+    mail (2.5.4)
+      mime-types (~> 1.16)
+      treetop (~> 1.4.8)
     method_source (0.8.1)
-    multi_json (1.7.6)
+    mime-types (1.23)
+    minitest (4.7.5)
+    multi_json (1.7.7)
     nokogiri (1.5.10)
+    polyglot (0.3.3)
+    pony (1.5)
+      mail (> 2.0)
     pry (0.9.12.2)
       coderay (~> 1.0.5)
       method_source (~> 0.8)
@@ -43,14 +57,19 @@ GEM
     rspec-mocks (2.13.1)
     ruby-prof (0.13.0)
     slop (3.4.5)
+    thread_safe (0.1.0)
+      atomic
     timecop (0.6.1)
+    treetop (1.4.14)
+      polyglot
+      polyglot (>= 0.3.1)
+    tzinfo (0.3.37)
     uuidtools (2.1.4)
 
 PLATFORMS
   ruby
 
 DEPENDENCIES
-  activesupport
   dynamo-autoscale!
   pry
   ripl

data/README.md

@@ -37,7 +37,26 @@ This project aims to take all of this into consideration and automatically scale
 your throughputs to enable you to deal with spikes and save money where
 possible.
 
-# Configuration
+# Usage
+
+First of all, you'll need to install this project as a gem:
+
+    $ gem install dynamo-autoscale
+
+This will give you access to the `dynamo-autoscale` executable. The executable
+takes a single argument, the path to a config file.
+
+## Configuration
+
+The configuration file is the central thing that `dynamo-autoscale` requires to
+function. It specifies what tables to monitor, maximum and minimum throughputs
+and where your ruleset is located.
+
+The `dynamo-autoscale` executable takes a single argument, and that is the path
+to the configuration file you want to use.
+
+**A sample config can be found in the project root directory.** It documents all
+of the options you can specify.
 
 This library requires AWS keys that have access to both CloudWatch and DynamoDB,
 for retriving data and sending scaling requests. Using IAM, create a new user, and
@@ -52,29 +71,30 @@ The ARN for the custom policy can be specified as '\*' to allow access to all ta
 or alternatively you can refer to the IAM documentation to limit access to specific
 tables only.
 
-The project will look for a YAML file in the following locations on start up:
-
-  - ./aws.yml
-  - ENV['AWS_CONFIG']
-
-If it doesn't find an AWS YAML config in any of those locations, the process
-prints an error and exits.
-
-**A sample config can be found in the project root directory.**
+### Minimal "getting started" configuration
 
-# Usage
+``` yaml
+:aws:
+  :access_key_id:      "your_id"
+  :secret_access_key:  "your_key"
+  :dynamo_db_endpoint: "dynamodb.us-east-1.amazonaws.com"
 
-First of all, you'll need to install this project as a gem:
+# There are some example rulesets in the rulesets/ directory of this project.
+:ruleset: "path_to_your_ruleset.rb"
 
-    $ gem install dynamo-autoscale
+:tables:
+  - "your_table_name"
 
-This will give you access to the `dynamo-autoscale` executable. For some
-internal documentation on the executable, you can run:
+# In dry-run mode, the program will do exactly what it would normally except it
+# won't touch DynamoDB at all. It will just log the changes it would have made
+# in production locally.
+:dry_run: true
+```
 
-    $ dynamo-autoscale -h
+Save this somewhere on your filesystem and point the `dynamo-autoscale`
+executable to it:
 
-This should tell you what flags you can set and what arguments the command
-expects.
+    $ dynamo-autoscale path/to/config.yml
 
 ## Logging
 
@@ -84,16 +104,8 @@ variable set to `true`:
 
     $ DEBUG=true dynamo-autoscale <args...>
 
-Also, if you want pretty coloured logging, you can set the `PRETTY_LOG`
-environment variable to `true`:
-
-    $ PRETTY_LOG=true DEBUG=true dynamo-autoscale <args...>
-
 ## Rulesets
 
-One of the first things you'll notice upon looking into the `--help` on the
-executable is that it's looking for a "rule set". What on earth is a rule set?
-
 A rule set is the primary user input for dynamo-autoscale. It is a DSL for
 specifying when to increase and decrease your provisioned throughputs. Here is a
 very basic rule set:
@@ -236,10 +248,13 @@ your four downscales for the current day. Or, you can downscale reads and writes
 at the same time and this also costs you one of your four. (Reference:
 http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html)
 
-Because of this, the actioner can handle the grouping up of downscales. Let's
-say you passed in the following options in at the command line:
+Because of this, the actioner can handle the grouping up of downscales by adding
+the following to your config:
 
-    $ dynamo-autoscale some/ruleset.rb some_table --group-downscales --flush-after 300
+``` yaml
+:group_downscales: true
+:flush_after: 300
+```
 
 What this is saying is that if a write downscale came in, the actioner wouldn't
 fire it off immediately. It would wait 300 seconds, or 5 minutes, to see if a
@@ -248,11 +263,11 @@ time. If no corresponding read came in, after 5 minutes the pending write
 downscale would get "flushed" and applied without a read downscale.
 
 This technique helps to save downscales on tables that may have unpredictable
-consumption. You may need to tweak the `--flush-after` value to match your own
-situation. By default, there is no `--flush-after` and downscales will wait
-indefinitely, this may not be desirable.
+consumption. You may need to tweak the `flush_after` value to match your own
+situation. By default, there is no `flush_after` and downscales will wait
+indefinitely, but this may not be desirable.
 
-## Signaling
+## Signalling
 
 The `dynamo-autoscale` process responds to the SIGUSR1 and SIGUSR2 signals. What
 we've done may be a dramatic bastardisation of what signals are intended for or
@@ -266,7 +281,7 @@ files in the directory it was run in.
 
 Example:
 
-    $ dynamo-autoscale some/ruleset.rb some_table
+    $ dynamo-autoscale path/to/config.yml
     # Runs as PID 1234. Wait for some time to pass...
     $ kill -USR1 1234
     $ cat some_table.csv
@@ -286,7 +301,34 @@ The CSV is in the following format:
 If you send SIGUSR2 to the process as it's running, the process will take all of
 the data it has on all of its tables and generate a graph for each table using R
 (see the Graphs section below). This is handy for visualising what the process
-is doing, especially after doing a few hours of a `--dry-run`.
+is doing, especially after doing a few hours of a `dry_run`.
+
+## Scale Report Emails
+
+If you would like to receive email notifications whenever a scale event happens,
+you can specify some email options in your configuration. Specifying the email
+options implicitly activates email reports. Not including your email config
+implicitly turns it off.
+
+Sample email config:
+
+``` yaml
+:email:
+  :to: "john.doe@example.com"
+  :from: "dynamo-autoscale@example.com"
+  :via: :smtp
+  :via_options:
+    :port: 25
+    :enable_starttls_auto: false
+    :authentication: :plain
+    :address: "mailserver.example.com"
+    :user_name: "user"
+    :password: "password"
+```
+
+We're using Pony internally to send email and this part of the config just gets
+passed to Pony verbatim. Check out the [Pony](https://github.com/benprew/pony)
+documentation for more details on the options it supports.
 
 # Developers / Tooling
 
@@ -333,25 +375,21 @@ with the `script/historic_data` executable.
 If you want to test rules on your local machine without having to query
 CloudWatch or hit DynamoDB, there are tools that facilitate that nicely.
 
-The first thing you would need to do is gather some historic data. There's a
-script called `script/historic_data` that you can run to gather data on a
-specific table and store it into the `data/` directory in a format that all of
-the other scripts are familiar with.
-
-Next there are a couple of things you can do.
-
 ### Running a test
 
 You can run a big batch of data all in one go with the `script/test` script.
 This script can be invoked like this:
 
-    $ script/test rulesets/default.rb table_name
+    $ script/test path/to/config.yml
+
+You will need to make sure you have historic data available for whatever tables
+you have listed in your config file. If you don't, it's easy to gather it:
 
-Substituting `table_name` with the name of a table that exists in your DynamoDB.
-This will run through all of the data for that table in time order, logging
-along the way and triggering rules from the rule set if any were defined.
+    $ script/historic_data path/to/config.yml
 
-At the end, it shows you a report on the amount of wasted, used and lost units.
+This script goes off to CloudWatch and pulls down about a week of data for each
+table you have listed in your config. Then you can continue to re-run the
+`script/test` command and watch a tonne of log output fly by.
 
 #### Graphs
 

data/bin/dynamo-autoscale

@@ -1,85 +1,24 @@
 #!/usr/bin/env ruby
 
-require 'pp'
-require 'optparse'
-require 'active_support/all'
-
-# Force this script into production mode as it's the only thing that will
-# actually hit DynamoDB in the entire project.
-ENV['RACK_ENV'] = "production"
-
-actioner_opts = {}
-general_opts  = {}
-
-OptionParser.new do |opts|
-  opts.banner = "Usage: dynamo-autoscale ruleset_path table_name [more table names] [options]"
-
-  doc = 'Makes read and write downscales happen at the same time to save ' +
-    'downscales per day.'
-
-  opts.on('-g', '--group-downscales', doc) do
-    actioner_opts[:group_downscales] = true
-  end
-
-  doc = 'Only works in conjunction with --group-downscales. Sets a maximum ' +
-    'amount of time for an operation to be pending before it gets applied to Dynamo'
-
-  opts.on('--flush-after SECONDS', Integer, doc) do |seconds|
-    actioner_opts[:flush_after] = seconds.to_i.seconds
-  end
-
-  doc = 'Stops dynamo-autoscale from talking to DynamoDB. Instead, it just ' +
-    'tracks the changes it would have made locally.'
-
-  opts.on('--dry-run', doc) do
-    general_opts[:dry_run] = true
-  end
-
-  doc = "Sets a minimum value for throughputs to be set to. " +
-    "Defaults to 10."
-
-  opts.on('--minimum-throughput VALUE', Float, doc) do |value|
-    if value < 1.0
-      STDERR.puts "Cannot set minimum throughput to less than 1."
-      exit 1
-    end
-
-    general_opts[:minimum_throughput] = value
-  end
-
-  doc = "Sets a maximum value for throughputs to be set to. " +
-    "Defaults to 20,000."
-
-  opts.on('--maximum-throughput VALUE', Float, doc) do |value|
-    general_opts[:maximum_throughput] = value
-  end
-
-  opts.on( '-h', '--help', 'Display this screen' ) do
-    puts opts
-    exit
-  end
-end.parse!
+require_relative '../config/environment/common'
 
-ruleset = ARGV.shift
-tables  = ARGV
+if ARGV[0]
+  DynamoAutoscale.setup_from_config(ARGV[0])
+elsif ARGV[0].nil?
+  STDERR.puts "Usage: dynamo-autoscale path/to/config.yml"
 
-if tables.empty? or ruleset.nil?
-  STDERR.puts "Usage: dynamo-autoscale ruleset table_name [another_table_name ... ]"
   exit 1
-end
+elsif ARGV[0] and !File.exists?(ARGV[0])
+  STDERR.puts "Usage: dynamo-autoscale path/to/config.yml"
+  STDERR.puts "Error: The path you specified is to a file that does not exist."
 
-if actioner_opts[:flush_after] and actioner_opts[:group_downscales].nil?
-  STDERR.puts "Cannot specify a flush_after value with setting --group-downscales."
   exit 1
 end
 
-require_relative '../config/environment/common'
-include DynamoAutoscale
-extend  DynamoAutoscale
-
-logger.debug "Ensuring tables exist in DynamoDB..."
+DynamoAutoscale.logger.info "Ensuring tables exist in DynamoDB..."
 dynamo = AWS::DynamoDB.new
-tables.select! do |table_name|
+
+DynamoAutoscale.poller_opts[:tables].select! do |table_name|
   if dynamo.tables[table_name].exists?
     true
   else
@@ -88,48 +27,11 @@ tables.select! do |table_name|
   end
 end
 
-if tables.empty?
-  STDERR.puts "No valid tables specified."
-  exit 1
-end
-
-poller_opts = { tables: tables }
-
-if general_opts[:dry_run]
-  poller_opts[:filters] = LocalActioner.faux_provisioning_filters
-end
-
-logger.debug "Assigning global objects..."
-DynamoAutoscale.rules          = RuleSet.new(ruleset)
-DynamoAutoscale.dispatcher     = Dispatcher.new
-DynamoAutoscale.poller         = CWPoller.new(poller_opts)
-DynamoAutoscale.actioner_class = general_opts[:dry_run] ? LocalActioner : DynamoActioner
-DynamoAutoscale.actioner_opts  = actioner_opts
+DynamoAutoscale.poller_class = DynamoAutoscale::CWPoller
 
-if general_opts[:minimum_throughput]
-  Actioner.minimum_throughput = general_opts[:minimum_throughput]
-end
-
-if general_opts[:maximum_throughput]
-  Actioner.maximum_throughput = general_opts[:maximum_throughput]
-end
-
-logger.debug "Registering signal handlers..."
-Signal.trap("USR1") do
-  logger.info "[signal] Caught SIGUSR1. Dumping CSV for all tables in #{Dir.pwd}"
-
-  DynamoAutoscale.tables.each do |name, table|
-    table.to_csv! path: File.join(Dir.pwd, "#{table.name}.csv")
-  end
-end
-
-Signal.trap("USR2") do
-  logger.info "[signal] Caught SIGUSR2. Dumping graphs for all tables in #{Dir.pwd}"
-
-  DynamoAutoscale.tables.each do |name, table|
-    table.graph! path: File.join(Dir.pwd, "#{table.name}.png")
-  end
+unless DynamoAutoscale.config[:dry_run]
+  DynamoAutoscale.actioner_class = DynamoAutoscale::DynamoActioner
 end
 
-logger.info "Finished setup. Starting polling loop."
+DynamoAutoscale.logger.info "Finished setup. Starting polling loop."
 DynamoAutoscale.poller.run

data/config/dynamo-autoscale-test.yml

@@ -0,0 +1,11 @@
+# A minimal config file for use in the testing suite.
+
+:logger:
+  :level: INFO
+  :sync:  true
+  :style: pretty
+
+:tables:
+  - "just keeping the config loader happy"
+
+:dry_run: true # fakes the provisioning data

data/config/environment/common.rb

@@ -1,4 +1,7 @@
 require 'logger'
+require 'optparse'
+require 'fileutils'
+require 'timecop'
 require 'time'
 require 'csv'
 require 'tempfile'
@@ -6,6 +9,9 @@ require 'aws-sdk'
 require 'active_support/all'
 require 'rbtree'
 require 'colored'
+require 'pp'
+require 'erb'
+require 'pony'
 
 require_relative '../../lib/dynamo-autoscale/logger'
 require_relative '../../lib/dynamo-autoscale/poller'
@@ -24,28 +30,77 @@ module DynamoAutoscale
     File.join(self.root, 'data')
   end
 
-  def self.env
-    ENV['RACK_ENV'] || 'development'
+  def self.config
+    @@config ||= {}
   end
 
-  def self.with_config name, opts ={}
-    path = nil
+  def self.config= new_config
+    @@config = new_config
+  end
+
+  def self.setup_from_config path, overrides = {}
+    logger.debug "[setup] Loading config..."
+    self.config = YAML.load_file(path).merge(overrides)
+
+    if config[:tables].nil? or config[:tables].empty?
+      STDERR.puts "You need to specify at least one table in your config's " +
+        ":tables section."
 
-    if opts[:absolute]
-      path = name
+      exit 1
+    end
+
+    filters = config[:dry_run] ? DynamoAutoscale::LocalActioner.faux_provisioning_filters : []
+    if filters.empty?
+      logger.debug "[setup] Not running as a dry run. Hitting production Dynamo."
     else
-      path = File.join(root, 'config', "#{name}.yml")
+      logger.debug "[setup] Running as dry run. No throughputs will be changed."
+    end
+
+    DynamoAutoscale.poller_opts = {
+      tables: config[:tables],
+      filters: filters,
+    }
+
+    logger.debug "[setup] Poller options are: #{DynamoAutoscale.poller_opts}"
+
+    DynamoAutoscale.actioner_opts = {
+      group_downscales: config[:group_downscales],
+      flush_after: config[:flush_after],
+    }
+
+    logger.debug "[setup] Actioner options are: #{DynamoAutoscale.actioner_opts}"
+
+    if config[:minimum_throughput]
+      DynamoAutoscale::Actioner.minimum_throughput = config[:minimum_throughput]
     end
 
-    conf = YAML.load_file(path)[env]
-    yield conf if block_given?
-    conf
+    if config[:maximum_throughput]
+      DynamoAutoscale::Actioner.maximum_throughput = config[:maximum_throughput]
+    end
+
+    logger.debug "[setup] Minimum throughput set to: " +
+      "#{DynamoAutoscale::Actioner.minimum_throughput}"
+    logger.debug "[setup] Maximum throughput set to: " +
+      "#{DynamoAutoscale::Actioner.maximum_throughput}"
+
+    logger.debug "[setup] Ruleset loading from: #{config[:ruleset]}"
+    DynamoAutoscale.ruleset_location = config[:ruleset]
+
+    logger.debug "[setup] Loaded #{DynamoAutoscale.rules.rules.values.flatten.count} rules."
+
+    DynamoAutoscale.load_services
   end
 
   def self.require_all path
     Dir[File.join(root, path, '*.rb')].each { |file| require file }
   end
 
+  def self.load_services
+    Dir[File.join(DynamoAutoscale.root, 'config', 'services', '*.rb')].each do |path|
+      load path
+    end
+  end
+
   def self.dispatcher= new_dispatcher
     @@dispatcher = new_dispatcher
   end
@@ -54,12 +109,28 @@ module DynamoAutoscale
     @@dispatcher ||= Dispatcher.new
   end
 
+  def self.poller_opts= new_poller_opts
+    @@poller_opts = new_poller_opts
+  end
+
+  def self.poller_opts
+    @@poller_opts ||= {}
+  end
+
+  def self.poller_class= new_poller_class
+    @@poller_class = new_poller_class
+  end
+
+  def self.poller_class
+    @@poller_class ||= LocalDataPoll
+  end
+
   def self.poller= new_poller
     @@poller = new_poller
   end
 
   def self.poller
-    @@poller ||= LocalDataPoll.new
+    @@poller ||= poller_class.new(poller_opts)
   end
 
   def self.actioner_class= klass
@@ -79,37 +150,41 @@ module DynamoAutoscale
   end
 
   def self.actioners
-    @@actioners ||= Hash.new { |h, k| h[k] = actioner_class.new(k, actioner_opts) }
+    @@actioners ||= Hash.new do |h, k|
+      h[k] = actioner_class.new(k, actioner_opts)
+    end
   end
 
-  def self.tables= new_tables
-    @@tables = new_tables
+  def self.reset_tables
+    @@tables = Hash.new { |h, k| h[k] = TableTracker.new(k) }
   end
 
   def self.tables
     @@tables ||= Hash.new { |h, k| h[k] = TableTracker.new(k) }
   end
 
-  def self.current_table= new_current_table
-    @@current_table = new_current_table
+  def self.ruleset_location
+    @@ruleset_location ||= nil
   end
 
-  def self.current_table
-    @@current_table
+  def self.ruleset_location= new_ruleset_location
+    @@ruleset_location = new_ruleset_location
   end
 
-  def self.rules= new_rules
-    @@rules = new_rules
+  def self.rules
+    @@rules ||= RuleSet.new(ruleset_location)
   end
 
-  def self.rules
-    @@rules ||= RuleSet.new
+  def self.current_table= new_current_table
+    @@current_table = new_current_table
+  end
+
+  def self.current_table
+    @@current_table
   end
 end
 
 DynamoAutoscale.require_all 'lib/dynamo-autoscale'
 DynamoAutoscale.require_all 'lib/dynamo-autoscale/ext/**'
 
-Dir[File.join(DynamoAutoscale.root, 'config', 'services', '*.rb')].each do |path|
-  load path
-end
+DynamoAutoscale.load_services

data/config/environment/test.rb

@@ -1,3 +1,6 @@
 ENV['RACK_ENV'] = "test"
 require_relative 'common'
 require 'timecop'
+
+path = File.join(DynamoAutoscale.root, 'config', 'dynamo-autoscale-test.yml')
+DynamoAutoscale.setup_from_config(path)

data/config/services/aws.rb

@@ -1,20 +1,3 @@
-config_location = nil
-
-if File.exists? './aws.yml'
-  config_location = './aws.yml'
-elsif ENV['AWS_CONFIG'] and File.exists? ENV['AWS_CONFIG']
-  config_location = ENV['AWS_CONFIG']
-elsif File.exists?(File.join(DynamoAutoscale.root, 'config', 'aws.yml'))
-  config_location = File.join(DynamoAutoscale.root, 'config', 'aws.yml')
-end
-
-if config_location.nil?
-  STDERR.puts "Could not load AWS configuration. Searched in: ./aws.yml and " +
-    "ENV['AWS_CONFIG']"
-
-  exit 1
-end
-
-DynamoAutoscale.with_config(config_location, absolute: true) do |config|
-  AWS.config(config)
+if DynamoAutoscale.config[:aws]
+  AWS.config(DynamoAutoscale.config[:aws])
 end