vector 0.0.1

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in vector.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Instructure, Inc
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,177 @@
+# Vector
+
+Vector is a tool that augments your auto-scaling groups. The two
+features currently offered are Predictive Scaling and Flexible Down
+Scaling.
+
+## Predictive scaling
+
+Auto Scaling groups do a good job of responding to current
+load conditions, but if you have a predictable load pattern,
+it can be nice to scale up your servers a little bit *early*.
+Some reasons you might want to do that are:
+
+ * If it takes several minutes for an instance to fully boot
+   and ready itself for requests.
+ * If you have very serious (but predictable) spikes,
+   it's nice to have the capacity in place before the spike
+   starts.
+ * To give yourself a buffer of time if AWS APIs start
+   throwing errors. If scaling up is going to fail, you'd
+   rather it start failing with a little bit of time before
+   you actually need the capacity so you can begin evasive maneuvers.
+
+Vector examines your existing CloudWatch alarms tied to your Auto
+Scaling groups, and predicts if they will be triggered in the future
+based on what happened in the past.
+
+**Note:** This only works with metrics that are averaged across your group -
+like CPUUtilization or Load. If you auto-scale based on something
+like QueueLength, Predictive Scaling will not work right for you.
+
+For each lookback window you specify, Vector will first check the
+current value of the metric * the number of nodes, and the past value of
+the metric * the past number of nodes. If those numbers are close enough
+(within the threshold specified by `--ps-valid-threshold`), then it will
+continue.
+
+Vector will then go back to the lookback window specified, and then
+forward in time based on the lookahead window (`--ps-lookahead-window`).
+It will compute the metric * number of nodes then to get a predicted
+aggregate metric value for the current future. It then divides that by
+the current number of nodes to get a predicted average value for the
+metric. That is then compared against the alarm's threshold.
+
+For example:
+
+> You have an alarm that checks CPUUtilization of your group, and will
+> trigger the alarm if that goes above 70%. Vector is configured to use a
+> 1 week lookback window, a 1 hour lookahead window, and a valid-threshold
+> of 0.8.
+> 
+> The current value of CPUUtilization is 49%, and there are 2 nodes in the
+> group. CPUUtilization 1 week ago was 53%, and there were 2 nodes in the
+> group. Therefore, total current CPUUtilization is 98%, and 1 week ago was
+> 106%. Those are within 80% of each other (valid-threshold), so we can
+> continue with the prediction.
+> 
+> The value of CPUUtilization 1 week ago, *plus* 1 hour was 45%, and
+> there were 4 nodes in the group. We calculate total CPUUtilization for
+> that time to be 180%. Assuming no new nodes are launched, the predicted
+> average CPUUtilization for the group 1 hour from now is 180% / 2 = 90%.
+> 90% is above the alarm's 75% threshold, so we trigger the scaleup
+> policy.
+
+
+If you use Predictive Scaling, you probably also want to use Flexible
+Down Scaling (below) so that after scaling up in prediction of load,
+your scaledown policy doesn't quickly undo Vector's hard work. You
+probably want to set `up-to-down-cooldown` to be close to the size of
+your `lookahead-window`.
+
+## Flexible Down Scaling
+
+### Different Cooldown Periods
+
+Auto Scaling Groups support the concept of "cooldown periods" - a window
+of time after a scaling activity where no other activities should take
+place. This is to give the group a chance to settle into the new
+configuration before deciding whether another action is required.
+
+However, Auto Scaling Groups only support specifying the cooldown period
+*after* a certain activity - you can say "After a scale up, wait 5
+minutes before doing anything else, and after a scale down, wait 15
+minutes." What you can't do is say "After a scale up, wait 5 minutes for
+another scale up, and 40 minutes for a scale down."
+
+Vector lets you add custom up-to-down and down-to-down cooldown periods.
+You create your policies and alarms in your Auto Scaling Groups like
+normal, and then *disable* the alarm you want a custom cooldown period
+applied to. Then you tell Vector what cooldown periods to use, and he
+does the rest.
+
+### Multiple Alarms
+
+Another benefit to Flexible Down Scaling is the ability to specify
+multiple alarms for a scaling down policy and require *all* alarms to
+trigger before scaling down. With Vector, you can add multiple
+(disabled) alarms to a policy, and Vector will trigger the policy only
+when *both* alarms are in ALARM state. This lets you do something like
+"only scale down when CPU utilization is < 30% and there is not a
+backlog of requests on any instances".
+
+### Max Sunk Cost
+
+Vector also lets you specify a "max sunk cost" when scaling down a node.
+Amazon bills on hourly increments, and you pay a full hour for every
+partial hour used, so you want your instances to terminate as close to
+their hourly billing renewal (without going past it).
+
+For example, if you specify `--fds-max-sunk-cost 15m` and have two nodes
+in your group - 47 minutes and 32 minutes away from their hourly billing
+renewals - the group will not be scaled down.
+
+(You should make sure to run Vector on an interval smaller than this
+one, or else it's possible Vector may never find eligible nodes for
+scaledown and never scaledown.)
+
+## Requirements
+
+ * Auto Scaling groups must have the GroupInServiceInstances metric
+   enabled.
+ * Auto Scaling groups must have at least one scaling policy with a
+   positive adjustment, and that policy must have at least one
+   CloudWatch alarm with a CPUUtilization metric.
+
+## Installation
+
+```bash
+$ gem install vector
+```
+
+## Usage
+
+Typically vector will be invoked via cron periodically (every 10 minutes
+is a good choice.)
+
+```
+Usage: vector [options]
+DURATION can look like 60s, 1m, 5h, 7d, 1w
+        --timezone TIMEZONE          Timezone to use for date calculations (like America/Denver) (default: system timezone)
+        --region REGION              AWS region to operate in (default: us-east-1)
+        --groups group1,group2       A list of Auto Scaling Groups to evaluate
+        --fleet fleet                An AWS ASG Fleet (instead of specifying --groups)
+    -v, --[no-]verbose               Run verbosely
+
+Predictive Scaling Options
+        --[no-]ps                    Enable Predictive Scaling
+        --ps-lookback-windows DURATION,DURATION
+                                     List of lookback windows
+        --ps-lookahead-window DURATION
+                                     Lookahead window
+        --ps-valid-threshold FLOAT   A number from 0.0 - 1.0 specifying how closely previous load must match current load for Predictive Scaling to take effect
+        --ps-valid-period DURATION   The period to use when doing the threshold check
+
+Flexible Down Scaling Options
+        --[no-]fds                   Enable Flexible Down Scaling
+        --fds-up-to-down DURATION    The cooldown period between up and down scale events
+        --fds-down-to-down DURATION  The cooldown period between down and down scale events
+        --fds-max-sunk-cost DURATION Only let a scaledown occur if there is an instance this close to its hourly billing point
+```
+
+# Questions
+
+### Why not just predictively scale based on the past DesiredInstances?
+
+If we don't look at the actual utilization and just look
+at how many instances we were running in the past, we will end up
+scaling earlier and earlier, and will never re-adjust and not scale up
+if load patterns change, and we don't need so much capacity.
+
+### What about high availability? What if the box Vector is running on dies?
+
+Luckily Vector is just providing optimizations - the critical component
+of scaling up based on demand is still provided by the normal Auto
+Scaling service. If Vector does not run, you just don't get the
+predictive scaling and down scaling.
+

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/bin/vector

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+
+if $0 == "bin/vector"
+  $LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__)) + "/../lib")
+end
+
+require 'vector'
+Vector::CLI.new(ARGV).run

data/lib/vector.rb

@@ -0,0 +1,82 @@
+require 'aws-sdk'
+require 'active_support/time'
+
+require 'vector/version'
+
+module Vector
+  def self.time_string_to_seconds(string)
+    if string =~ /^(\d+)([smhdw])?$/
+      n = $1.to_i
+      unit = $2 || 's'
+
+      case unit
+      when 's'
+        n.seconds
+      when 'm'
+        n.minutes
+      when 'h'
+        n.hours
+      when 'd'
+        n.days
+      when 'w'
+        n.weeks
+      end
+    else
+      nil
+    end
+  end
+
+  def self.within_threshold(threshold, v1, v2)
+    threshold * v1 < v2 && threshold * v2 < v1
+  end
+
+  module HLogger
+    @@enabled = false
+    def self.enable(bool)
+      @@enabled = bool
+    end
+
+    def hlog_ctx(ctx, &block)
+      @components ||= []
+      @components << ctx
+      yield
+    ensure
+      @components.pop
+    end
+
+    def hlog(string)
+      return unless @@enabled
+
+      tmp_components = @components.dup
+      level = 0
+      if @last_components
+        @last_components.each do |last_c|
+          break if tmp_components.empty?
+          if last_c == tmp_components[0]
+            level += 1
+            tmp_components.shift
+          else
+            break
+          end
+        end
+      end
+
+      tmp_components.each do |component|
+        name = if component.respond_to? :name
+                 component.name
+               else
+                 component.to_s
+               end
+        puts "#{"  " * level}#{name}"
+        level += 1
+      end
+
+      puts "#{"  " * level}#{string}"
+      @last_components = @components.dup
+    end
+  end
+end
+
+require 'vector/cli'
+require 'vector/functions/predictive_scaling'
+require 'vector/functions/flexible_down_scaling'

data/lib/vector/cli.rb

@@ -0,0 +1,193 @@
+require 'optparse'
+require 'aws-sdk'
+require 'aws/auto_scaling/fleets'
+require 'vector/functions/flexible_down_scaling'
+require 'vector/functions/predictive_scaling'
+
+module Vector
+  class CLI
+    def initialize(argv)
+      @argv = argv
+    end
+
+    def run
+      load_config
+
+      auto_scaling = AWS::AutoScaling.new(:region => @config[:region])
+      cloudwatch = AWS::CloudWatch.new(:region => @config[:region])
+
+      # everything we do should be fine looking at a snapshot in time,
+      # so memoizing should be fine when acting as a CLI.
+      AWS.start_memoizing
+
+      groups = if @config[:fleet]
+                 auto_scaling.fleets[@config[:fleet]].groups
+               else
+                 @config[:groups].map do |group_name|
+                   auto_scaling.groups[group_name]
+                 end
+               end
+
+      if @config[:predictive_scaling][:enabled]
+        psconf = @config[:predictive_scaling]
+
+        ps = Vector::Function::PredictiveScaling.new(
+          { :cloudwatch => cloudwatch }.merge(psconf))
+
+        groups.each do |group|
+          begin
+            ps.run_for(group)
+          rescue => e
+            puts "error performing Predictive Scaling on #{group.name}: #{e.inspect}\n#{e.backtrace.join "\n"}"
+          end
+        end
+      end
+
+      if @config[:flexible_down_scaling][:enabled]
+        fdsconf = @config[:flexible_down_scaling]
+
+        fds = Vector::Function::FlexibleDownScaling.new(
+          { :cloudwatch => cloudwatch }.merge(fdsconf))
+
+        groups.each do |group|
+          fds.run_for(group)
+        end
+      end
+    end
+
+    protected
+
+    def load_config
+      opts = {
+        :quiet => false,
+        :timezone => nil,
+        :region => 'us-east-1',
+        :groups => [],
+        :fleet => nil,
+        :predictive_scaling => {
+          :enabled => false,
+          :lookback_windows => [],
+          :lookahead_window => nil,
+          :valid_threshold => nil,
+          :valid_period => 60 * 10
+        },
+        :flexible_down_scaling => {
+          :enabled => false,
+          :up_down_cooldown => nil,
+          :down_down_cooldown => nil,
+          :max_sunk_cost => nil
+        }
+      }
+
+      optparser = OptionParser.new do |o|
+        o.banner = "Usage: vector [options]"
+        o.separator "DURATION can look like 60s, 1m, 5h, 7d, 1w"
+
+        o.on("--timezone TIMEZONE", "Timezone to use for date calculations (like America/Denver) (default: system timezone)") do |v|
+          opts[:timezone] = v
+        end
+
+        o.on("--region REGION", "AWS region to operate in (default: us-east-1)") do |v|
+          opts[:region] = v
+        end
+
+        o.on("--groups group1,group2", Array, "A list of Auto Scaling Groups to evaluate") do |v|
+          opts[:groups] = v
+        end
+
+        o.on("--fleet fleet", "An AWS ASG Fleet (instead of specifying --groups)") do |v|
+          opts[:fleet] = v
+        end
+
+        o.on("-q", "--[no-]quiet", "Run quietly") do |v|
+          opts[:quiet] = v
+        end
+
+        o.separator ""
+        o.separator "Predictive Scaling Options"
+
+        o.on("--[no-]ps", "Enable Predictive Scaling") do |v|
+          opts[:predictive_scaling][:enabled] = v
+        end
+
+        o.on("--ps-lookback-windows DURATION,DURATION", Array, "List of lookback windows") do |v|
+          opts[:predictive_scaling][:lookback_windows] =
+            v.map {|w| Vector.time_string_to_seconds(w) }
+        end
+
+        o.on("--ps-lookahead-window DURATION", String, "Lookahead window") do |v|
+          opts[:predictive_scaling][:lookahead_window] =
+            Vector.time_string_to_seconds(v)
+        end
+
+        o.on("--ps-valid-threshold FLOAT", Float, "A number from 0.0 - 1.0 specifying how closely previous load must match current load for Predictive Scaling to take effect") do |v|
+          opts[:predictive_scaling][:valid_threshold] = v
+        end
+
+        o.on("--ps-valid-period DURATION", String, "The period to use when doing the threshold check") do |v|
+          opts[:predictive_scaling][:valid_period] = 
+            Vector.time_string_to_seconds v
+        end
+
+        o.separator ""
+        o.separator "Flexible Down Scaling Options"
+
+        o.on("--[no-]fds", "Enable Flexible Down Scaling") do |v|
+          opts[:flexible_down_scaling][:enabled] = v
+        end
+
+        o.on("--fds-up-to-down DURATION", String, "The cooldown period between up and down scale events") do |v|
+          opts[:flexible_down_scaling][:up_down_cooldown] =
+            Vector.time_string_to_seconds v
+        end
+
+        o.on("--fds-down-to-down DURATION", String, "The cooldown period between down and down scale events") do |v|
+          opts[:flexible_down_scaling][:down_down_cooldown] =
+            Vector.time_string_to_seconds v
+        end
+
+        o.on("--fds-max-sunk-cost DURATION", String, "Only let a scaledown occur if there is an instance this close to its hourly billing point") do |v|
+          time = Vector.time_string_to_seconds v
+          if time > 1.hour
+            puts "--fds-max-sunk-cost duration must be < 1 hour"
+            exit 1
+          end
+
+          opts[:flexible_down_scaling][:max_sunk_cost] = time
+        end
+
+      end.parse!(@argv)
+
+      if opts[:groups].empty? && opts[:fleet].nil?
+        puts "No groups were specified."
+        exit 1
+      end
+
+      if !opts[:groups].empty? && !opts[:fleet].nil?
+        puts "You can't specify --groups and --fleet."
+        exit 1
+      end
+
+      if opts[:predictive_scaling][:enabled]
+        ps = opts[:predictive_scaling]
+        if ps[:lookback_windows].empty? || ps[:lookahead_window].nil?
+          puts "You must specify lookback windows and a lookahead window for Predictive Scaling."
+          exit 1
+        end
+      end
+
+      if opts[:flexible_down_scaling][:enabled]
+        fds = opts[:flexible_down_scaling]
+        if fds[:up_down_cooldown].nil? ||
+           fds[:down_down_cooldown].nil?
+          puts "You must specify both up-to-down and down-to-down cooldown periods for Flexible Down Scaling."
+          exit 1
+        end
+      end
+
+      Vector::HLogger.enable(!opts[:quiet])
+
+      @config = opts
+    end
+  end
+end

data/lib/vector/functions/flexible_down_scaling.rb

@@ -0,0 +1,187 @@
+require 'vector'
+
+module Vector
+  module Function
+    class FlexibleDownScaling
+      include Vector::HLogger
+
+      def initialize(options)
+        @cloudwatch = options[:cloudwatch]
+        @up_down_cooldown = options[:up_down_cooldown]
+        @down_down_cooldown = options[:down_down_cooldown]
+        @max_sunk_cost = options[:max_sunk_cost]
+      end
+
+      def run_for(group)
+        hlog_ctx("group: #{group.name}") do
+          # don't check if no config was specified
+          if @up_down_cooldown.nil? && @down_down_cooldown.nil?
+            hlog("No cooldown periods specified, exiting")
+            return nil
+          end
+
+          # don't bother checking for a scaledown if desired capacity is
+          # already at the minimum size...
+          if group.desired_capacity == group.min_size
+            hlog("Group is already at minimum size, exiting")
+            return nil
+          end
+
+          scaledown_policies = group.scaling_policies.select do |policy|
+            policy.scaling_adjustment < 0
+          end
+
+          scaledown_policies.each do |policy|
+            hlog_ctx("policy: #{policy.name}") do
+              alarms = policy.alarms.keys.map do |alarm_name|
+                @cloudwatch.alarms[alarm_name]
+              end
+
+              # only consider disabled alarms (enabled alarms will trigger
+              # the policy automatically)
+              disabled_alarms = alarms.select do |alarm|
+                !alarm.enabled?
+              end
+
+              unless disabled_alarms.all? {|alarm| alarm.state_value == "ALARM" }
+                hlog("Not all alarms are in ALARM state")
+                next
+              end
+
+              unless outside_cooldown_period(group)
+                hlog("Group is not outside the specified cooldown periods")
+                next
+              end
+
+              unless has_eligible_scaledown_instance(group)
+                hlog("Group does not have an instance eligible for scaledown due to max_sunk_cost")
+                next
+              end
+
+              hlog("Executing policy")
+              policy.execute(:honor_cooldown => true)
+
+              # no need to evaluate other scaledown policies
+              return
+            end
+          end
+        end
+      end
+
+      protected
+
+      def has_eligible_scaledown_instance(group)
+        return true if @max_sunk_cost.nil?
+
+        group.ec2_instances.select {|i| i.status == :running }.each do |instance|
+          # get amount of time until hitting the instance renewal time
+          time_left = ((instance.launch_time.min - Time.now.min) % 60).minutes
+
+          # if we're within 1 minute, assume we won't be able to terminate it
+          # in time anyway and ignore it.
+          if time_left > 1.minute and time_left < @max_sunk_cost
+            # we only care if there is at least one instance within the window
+            # where we can scale down
+            return true
+          end
+        end
+
+        false
+      end
+
+      def outside_cooldown_period(group)
+        @cached_outside_cooldown ||= {}
+        if @cached_outside_cooldown.has_key? group
+          return @cached_outside_cooldown[group]
+        end
+
+        activities = previous_scaling_activities(group)
+        return nil if activities.nil?
+
+        if activities[:up]
+          hlog "Last scale up #{(Time.now - activities[:up]).minutes.inspect} ago"
+        end
+        if activities[:down]
+          hlog "Last scale down #{(Time.now - activities[:down]).minutes.inspect} ago"
+        end
+        result = true
+
+        # check up-down
+        if @up_down_cooldown && activities[:up] &&
+          Time.now - activities[:up] < @up_down_cooldown
+          result = false
+        end
+
+        # check down-down
+        if @down_down_cooldown && activities[:down] &&
+          Time.now - activities[:down] < @down_down_cooldown
+          result = false
+        end
+
+        result
+      end
+
+      # Looks at the GroupDesiredCapacity metric for the specified
+      # group, and finds the most recent change in value.
+      #
+      # @returns
+      #   * nil if there was a problem getting data. There may have been
+      #     scaling events or not, we don't know.
+      #   * a hash with two keys, :up and :down, with values indicating
+      #     when the last corresponding activity happened. If the
+      #     activity was not seen in the examined time period, the value
+      #     is nil.
+      def previous_scaling_activities(group)
+        metric = @cloudwatch.metrics.
+          with_namespace("AWS/AutoScaling").
+          with_metric_name("GroupDesiredCapacity").
+          filter('dimensions', [{
+            :name => "AutoScalingGroupName",
+            :value => group.name
+          }]).first
+
+        return nil unless metric
+
+        start_time = Time.now - [ @up_down_cooldown, @down_down_cooldown ].max
+        end_time = Time.now
+
+        stats = metric.statistics(
+          :start_time => start_time,
+          :end_time => end_time,
+          :statistics => [ "Average" ],
+          :period => 60)
+
+        # check if we got enough datapoints... if we didn't, we need to
+        # assume bad data and inform the caller. this code is basically
+        # checking if the # of received datapoints is within 50% of the
+        # expected datapoints.
+        got_datapoints = stats.datapoints.length
+        requested_datapoints = (end_time - start_time) / 60
+        if !Vector.within_threshold(0.5, got_datapoints, requested_datapoints)
+          return nil
+        end
+
+        # iterate over the datapoints in reverse, looking for the first
+        # change in value, which should be the most recent scaling
+        # activity
+        activities = { :down => nil, :up => nil }
+        last_value = nil
+        stats.datapoints.sort {|a,b| b[:timestamp] <=> a[:timestamp] }.each do |dp|
+          next if dp[:average].nil?
+
+          unless last_value.nil?
+            if dp[:average] != last_value
+              direction = (last_value < dp[:average]) ? :down : :up
+              activities[direction] ||= dp[:timestamp]
+            end
+          end
+
+          last_value = dp[:average]
+          break unless activities.values.any? {|v| v.nil? }
+        end
+
+        activities
+      end
+    end
+  end
+end

data/lib/vector/functions/predictive_scaling.rb

@@ -0,0 +1,150 @@
+module Vector
+  module Function
+    class PredictiveScaling
+      include Vector::HLogger
+
+      def initialize(options)
+        @cloudwatch = options[:cloudwatch]
+        @lookback_windows = options[:lookback_windows]
+        @lookahead_window = options[:lookahead_window]
+        @valid_threshold = options[:valid_threshold]
+        @valid_period = options[:valid_period]
+      end
+
+      def run_for(group)
+        hlog_ctx "group: #{group.name}" do
+          return if @lookback_windows.length == 0
+
+          scaleup_policies = group.scaling_policies.select do |policy|
+            policy.scaling_adjustment > 0
+          end
+
+          scaleup_policies.each do |policy|
+            hlog_ctx "policy: #{policy.name}" do
+
+              policy.alarms.keys.each do |alarm_name|
+                alarm = @cloudwatch.alarms[alarm_name]
+                hlog_ctx "alarm: #{alarm.name} (metric #{alarm.metric.name})" do
+
+                  unless alarm.enabled?
+                    hlog "Skipping disabled alarm"
+                    next
+                  end
+
+                  # Note that everywhere we say "load" what we mean is
+                  # "metric value * number of nodes"
+                  now_load, now_num = load_for(group, alarm.metric,
+                    Time.now, @valid_period)
+
+                  if now_load.nil?
+                    hlog "Could not get current total for metric"
+                    next
+                  end
+
+                  @lookback_windows.each do |window|
+                    hlog_ctx "window: #{window.inspect}" do
+                      then_load, = load_for(group, alarm.metric,
+                        Time.now - window, @valid_period)
+
+                      if then_load.nil?
+                        hlog "Could not get past total value for metric"
+                        next
+                      end
+
+                      # check that the past total utilization is within
+                      # threshold% of the current total utilization
+                      if @valid_threshold &&
+                        !Vector.within_threshold(@valid_threshold, now_load, then_load)
+                        hlog "Past metric total value not within threshold (current #{now_load}, then #{then_load})"
+                        next
+                      end
+
+                      past_load, = load_for(group, alarm.metric,
+                        Time.now - window + @lookahead_window,
+                        alarm.period)
+
+                      if past_load.nil?
+                        hlog "Could not get past + #{@lookahead_window.inspect} total value for metric"
+                        next
+                      end
+
+                      # now take the past total load and divide it by the
+                      # current number of instances to get the predicted value
+                      predicted_value = past_load.to_f / now_num
+                      hlog "Predicted #{alarm.metric.name}: #{predicted_value}"
+
+                      if check_alarm_threshold(alarm, predicted_value)
+                        hlog "Executing policy"
+                        policy.execute(honor_cooldown: true)
+
+                        # don't need to evaluate further windows or policies on this group
+                        return
+                      end
+                    end
+                  end
+                end
+              end
+            end
+          end
+        end
+      end
+
+      protected
+
+      def check_alarm_threshold(alarm, value)
+        case alarm.comparison_operator
+        when "GreaterThanOrEqualToThreshold"
+          value >= alarm.threshold
+        when "GreaterThanThreshold"
+          value > alarm.threshold
+        when "LessThanThreshold"
+          value < alarm.threshold
+        when "LessThanOrEqualToThreshold"
+          value <= alarm.threshold
+        end
+      end
+
+      def load_for(group, metric, time, window)
+        num_instances_metric = @cloudwatch.metrics.
+          with_namespace("AWS/AutoScaling").
+          with_metric_name("GroupInServiceInstances").
+          filter('dimensions', [{
+            :name => 'AutoScalingGroupName',
+            :value => group.name
+          }]).first
+
+        unless num_instances_metric
+          raise "Could not find GroupInServicesInstances metric for #{group.name}"
+        end
+
+        start_time = time - (window / 2)
+        end_time = time + (window / 2)
+
+        avg = average_for_metric(metric, start_time, end_time)
+        num = average_for_metric(num_instances_metric, start_time, end_time)
+
+        if avg.nil? || num.nil?
+          return [ nil, nil ]
+        end
+
+        [ avg * num, num ]
+      end
+
+      def average_for_metric(metric, start_time, end_time)
+        stats = metric.statistics(
+          :start_time => start_time,
+          :end_time => end_time,
+          :statistics => [ "Average" ],
+          :period => 60)
+
+        return nil if stats.datapoints.length == 0
+
+        sum = stats.datapoints.inject(0) do |r, dp|
+          r + dp[:average]
+        end
+
+        sum.to_f / stats.datapoints.length
+      end
+    end
+  end
+end

data/lib/vector/version.rb

@@ -0,0 +1,3 @@
+module Vector
+  VERSION = "0.0.1"
+end

data/vector.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'vector/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "vector"
+  spec.version       = Vector::VERSION
+  spec.authors       = ["Zach Wily"]
+  spec.email         = ["zach@zwily.com"]
+  spec.summary       = %q{AWS Auto-Scaling Assistant}
+  spec.homepage      = "http://github.com/instructure/vector"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "aws-sdk"
+  spec.add_dependency "aws-asg-fleet"
+  spec.add_dependency "activesupport"
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+end

metadata

@@ -0,0 +1,140 @@
+--- !ruby/object:Gem::Specification
+name: vector
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Zach Wily
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-07-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: aws-asg-fleet
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- zach@zwily.com
+executables:
+- vector
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/vector
+- lib/vector.rb
+- lib/vector/cli.rb
+- lib/vector/functions/flexible_down_scaling.rb
+- lib/vector/functions/predictive_scaling.rb
+- lib/vector/version.rb
+- vector.gemspec
+homepage: http://github.com/instructure/vector
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: AWS Auto-Scaling Assistant
+test_files: []
+has_rdoc: