aws_tracker 0.0.1

data/.gitignore

@@ -0,0 +1,12 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*
+*~
+.DS_Store
+*.tmproj
+config/database.yml
+config/tracker_db.yml
+db/*.sqlite3
+log/*.log
+tmp/**/*

data/.rspec

@@ -0,0 +1 @@
+--colour

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in aws_tracker.gemspec
+gemspec

data/README.md

@@ -0,0 +1,111 @@
+AWS Tracker
+================
+Models the state of one or more Amazon Web Services accounts in an ActiveRecord object graph.
+
+  *ALPHA VERSION - not yet fully functional*
+
+
+----------------
+What is it?
+----------------
+The AWS Tracker monitors one or more AWS accounts and the current state of their associated "resources" -- Instances, EBS volumes, RDS servers, and so on. The Tracker periodically polls the Amazon Web Services API for the latest information on each account's resources, and then maintains the corresponding rows in an ActiveRecord-compatible database. Users of the library can thereafter run multiple complex queries on the state of their AWS assets, with no network overhead.
+
+
+----------------
+Why is it?
+----------------
+The AWS Tracker is intended to be a foundation library, on top of which more complex AWS applications can be built. It decouples the process of downloading AWS state information from the process of displaying and/or acting on that information. Although an executable 'tracker' command-line program is included, the library is primarily intended for use from within Rails, or some other ActiveRecord context. An ActiveRecord database connection must be initialized before the aws_tracker library is loaded from within Ruby.
+
+
+----------------
+Installation
+----------------
+1) Install the AWS Tracker gem (use _sudo_ as necessary).
+
+    gem install aws_tracker
+
+2) Install your database adaptor of choice. Sqlite3 is installed with AWS Tracker, so this step may be optional.
+
+
+----------------
+Usage [from the command line]
+----------------
+1) First, generate an ActiveRecord-style configuration file. 
+    
+  Here are the sample contents of `tracker_db.yml`:
+
+    adapter: sqlite3
+    database: aws_tracker.sqlite3
+    pool: 5
+    timeout: 5000    
+
+2) Create the database to contain the data. This is not necessary if using sqlite.
+
+3) To store your AWS account information in the database, export the credentials as environment variables. This is only needed the first time you run.
+
+    export AWS_ACCOUNT_NUMBER=[your AWS account number]
+    export AWS_ACCESS_KEY_ID=[your EC2 access key ID]
+    export AWS_ACCESS_KEY_SECRET=[your EC2 access key secret]
+    
+4) Run the tracker, and point it at the database config file
+
+    tracker tracker_db.yml --migrate --create-account
+
+  * The `--migrate` argument updates the database to the latest version of the schema, and is only necessary for new databases, or when upgrading to a new version of the Tracker gem.  
+  * The `--create-account` argument stores the above environment values as an AWSAccount object in the tracker database.
+
+
+----------------
+Usage [from within Ruby]
+----------------
+1) Insert the necessary tables into your database.
+  First, `require 'AWSTracker/tasks'` from your Rakefile, then run
+    
+    rake db:migrate:tracker
+
+2) In your Ruby app, first set up an ActiveRecord connection. In Rails, this is done for you automatically, but here's an example for a non-Rails app:
+
+    require 'active_record'
+    ActiveRecord::Base.establish_connection({
+      :adapter => 'sqlite3', :database => 'aws_tracker.sqlite3'
+    })
+
+3) Now `require 'aws_tracker'`. This will load the models and allow you to perform queries on the following AWS resources and their relationships:
+
+  * AWSTracker::AWSAccount
+  * AWSTracker::SecurityGroup
+  * AWSTracker::Instance
+  * AWSTracker::EBSVolume
+
+But first, you need to create an AWSAccount, and use a Tracker object to fetch its status from AWS:
+
+    include AWSTracker          # Load model class names into namespace
+    a = AWSAccount.from_memory  # Create an account (see command-line usage)
+    a.save!                     # Save it in the database
+    t = Tracker.new             # A Tracker updates all accounts in the DB
+    t.delay = 15                # Update every 15 seconds
+    t.start                     # Runs in the background. Call 'stop' later
+
+
+----------------
+Development
+----------------
+This project is still in its early stages, but most of the framework is in place. More AWS resources need to be modeled, but the patterns for the code to do so are now laid out. Helping hands are appreciated!
+
+1) Fetch the project code and bundle up...
+
+    git clone https://github.com/benton/aws_tracker.git
+    cd aws_tracker
+    bundle install
+
+2) Create a database config and the necessary tables
+
+    cp config/database.yml.example config/database.yml
+    rake db:migrate:tracker
+
+3) To run the tests, first export the credentials as environment variables, then `rake spec`.
+
+    export AWS_ACCOUNT_NUMBER=[your AWS account number]
+    export AWS_ACCESS_KEY_ID=[your EC2 access key ID]
+    export AWS_ACCESS_KEY_SECRET=[your EC2 access key secret]
+    rake spec

data/Rakefile

@@ -0,0 +1,9 @@
+require "rubygems"
+require 'bundler'
+Bundler.setup(:default, :test, :development)
+
+# Load all tasks from 'lib/tasks'
+Dir["#{File.dirname(__FILE__)}/lib/tasks/*.rake"].sort.each {|ext| load ext}
+
+desc 'Default: run specs.'
+task :default => :spec

data/aws_tracker.gemspec

@@ -0,0 +1,33 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "aws_tracker/version"
+
+Gem::Specification.new do |s|
+  s.name        = "aws_tracker"
+  s.version     = AwsTracker::VERSION
+  s.platform    = Gem::Platform::RUBY
+  s.authors     = ["Benton Roberts"]
+  s.email       = ["benton@bentonroberts.com"]
+  s.homepage    = "http://github.com/benton/aws_tracker"
+  s.summary     = %q{Models the state of Amazon Web Services accounts in ActiveRecord}
+  s.description = %q{This gem peridically queries AWS for information about }+
+                  %q{Instances, SecurityGroups, EBSVolumes, etc., and saves }+
+                  %q{the data as interconnected ActiveRecord objects.}+
+
+  s.rubyforge_project = "aws_tracker"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  # Runtime dependencies
+  s.add_dependency "activerecord"
+  s.add_dependency "right_aws"
+  s.add_dependency "sqlite3"
+
+  # Development / Test dependencies
+  s.add_development_dependency "rake"
+  s.add_development_dependency "rspec"
+
+end

data/bin/tracker

@@ -0,0 +1,104 @@
+#!/usr/bin/env ruby 
+
+# == Synopsis 
+#   Monitors one or more AWS accounts and pushes their status into a database.
+#
+# == Usage 
+#   tracker.rb [options] DB_CONFIG_FILE.YML
+#
+# == Options (all options can be put into the config file)
+#   -d, --delay [INTEGER]       Seconds between status updates. default = 180
+#   -l, --log-level [LEVEL]     Sets Log4r level for console output. default = INFO
+#   -m, --migrate               Update database schema
+#   -c, --create-account        Create a new Account from environment variables
+#   -h, --help                  Displays help message
+#
+# == Author
+#   Benton Roberts
+
+require 'logger'
+require 'optparse'
+require 'yaml'
+LOG_LVLS = {
+  "DEBUG" => ::Logger::DEBUG,
+  "INFO"  => ::Logger::INFO,
+  "WARN"  => ::Logger::WARN,
+  "ERROR" => ::Logger::ERROR,
+  "FATAL" => ::Logger::FATAL 
+}
+
+class AWSTrackerConsoleApp
+
+  def initialize
+    @log = ::Logger.new(STDOUT)
+    @log.formatter = proc {|lvl, time, prog, msg|
+      "#{lvl} #{time.strftime '%Y-%m-%d %H:%M:%S %Z'}: #{msg}\n"
+    }
+    parse_options
+    (@log.error "A database config file must be specified" ; exit) if ARGV.empty?
+    db_config_file = ARGV.first
+    @log.info "Loading database configuration from #{db_config_file}"
+    db_config = YAML::load(File.open(db_config_file))
+    connect_to_database(db_config)
+    create_account if @opts[:create_account]
+    go
+  end
+
+  def connect_to_database(db_config)
+    @log.info "Connecting to database #{db_config['database']}"
+    require 'active_record'
+    ::ActiveRecord::Base.establish_connection(db_config)
+    migrate if @opts[:migrate]
+    require 'aws_tracker'
+  end
+
+  def go
+    @tracker = AWSTracker::Tracker.new(:logger => @log)
+    @tracker.delay = @opts[:delay]
+    @tracker.start
+    while true do 
+      sleep @opts[:delay]   # Loop forever
+    end
+  end
+
+  def parse_options
+    @opts = { :delay  => 180, :log_level => 'INFO' }
+    optparse = OptionParser.new do |opts|
+      opts.on('-d', '--delay SECONDS', Integer,
+      'Number of seconds between status updates') do |delay|
+        @opts[:delay] = delay
+      end
+      opts.on('-l', '--log-level LEVEL', 'Set logging level') do |log_level|
+        @opts[:log_level] = log_level.upcase
+      end
+      opts.on('-m', '--migrate', 'Update database schema') do
+        @opts[:migrate] = true
+      end
+      acct_help = "Create a new AWSAccount object from ENV varaibles"
+      opts.on('-c', '--create-account', acct_help) do
+        @opts[:create_account] = true
+      end
+      opts.on('-h', '--help', 'Display this help message') do
+        puts opts and exit
+      end
+    end
+    optparse.parse!
+    @log.level = LOG_LVLS[@opts[:log_level]] if LOG_LVLS[@opts[:log_level]]
+  end
+
+  def migrate
+    @log.info "Updating database schema..."
+    migration_dir = File.expand_path('../../db/migrate', __FILE__)
+    ActiveRecord::Migrator.migrate migration_dir
+  end
+
+  def create_account
+    @log.info "Creating account from environment..."
+    a = AWSTracker::AWSAccount.from_memory
+    a.log.info "Loaded account #'#{a.account_id}' from memory"
+    a.save!
+  end
+
+end
+
+myApp = AWSTrackerConsoleApp.new

data/config/database.yml.example

@@ -0,0 +1,4 @@
+adapter: sqlite3
+database: db/aws_tracker.sqlite3
+pool: 5
+timeout: 5000

data/db/migrate/20110331145142_create_aws_accounts.rb

@@ -0,0 +1,16 @@
+class CreateAwsAccounts < ActiveRecord::Migration
+  def self.up
+    create_table :aws_accounts do |t|
+      t.string :name
+      t.string :account_id
+      t.string :access_key_id
+      t.string :secret_access_key
+
+      t.timestamps
+    end
+  end
+
+  def self.down
+    drop_table :aws_accounts
+  end
+end

data/db/migrate/20110331152735_create_instances.rb

@@ -0,0 +1,38 @@
+class CreateInstances < ActiveRecord::Migration
+  def self.up
+    create_table :instances do |t|
+      t.integer :aws_account_id
+      t.string :instance_id
+      t.string :image_id
+      t.string :instance_state
+      t.integer :state_code
+      t.string :private_dns_name
+      t.string :dns_name
+      t.string :key_name
+      t.integer :ami_launch_index
+      t.string :instance_type
+      t.datetime :launch_time
+      t.datetime :termination_time
+      t.string :availability_zone
+      t.string :kernel_id
+      t.string :ramdisk_id
+      t.string :private_ip_address
+      t.string :ip_address
+      t.string :architecture
+      t.string :root_device_type
+      t.string :root_device_name
+      t.string :instance_life_cycle
+      t.string :spot_instance_request_id
+      t.string :virtualization_type
+      t.string :reason
+      t.string :state_reason_message
+      t.string :vpc_id
+
+      t.timestamps
+    end
+  end
+
+  def self.down
+    drop_table :instances
+  end
+end

data/db/migrate/20110403041402_create_security_groups.rb

@@ -0,0 +1,16 @@
+class CreateSecurityGroups < ActiveRecord::Migration
+  def self.up
+    create_table :security_groups do |t|
+      t.integer :aws_account_id
+      t.string :name
+      t.string :description
+      t.string :owner
+
+      t.timestamps
+    end
+  end
+
+  def self.down
+    drop_table :security_groups
+  end
+end

data/db/migrate/20110403060000_create_instance_groups.rb

@@ -0,0 +1,14 @@
+class CreateInstanceGroups < ActiveRecord::Migration
+  def self.up
+    create_table :instances_security_groups, :id => false do |t|
+      t.integer :instance_id
+      t.integer :security_group_id
+
+#      t.timestamps
+    end
+  end
+
+  def self.down
+    drop_table :instances_security_groups
+  end
+end

data/db/migrate/20110404014218_create_ebs_volumes.rb

@@ -0,0 +1,23 @@
+class CreateEbsVolumes < ActiveRecord::Migration
+  def self.up
+    create_table :ebs_volumes do |t|
+      t.integer :aws_account_id
+      t.string :volume_id
+      t.integer :size
+      t.string :status
+      t.string :availability_zone
+      t.string :snapshot_id
+      t.datetime :create_time
+      t.datetime :attach_time
+      t.string :attachment_status
+      t.string :device
+      t.integer :instance_id
+
+      t.timestamps
+    end
+  end
+
+  def self.down
+    drop_table :ebs_volumes
+  end
+end

data/db/migrate/20110408193824_create_tags.rb

@@ -0,0 +1,17 @@
+class CreateTags < ActiveRecord::Migration
+  def self.up
+    create_table :tags do |t|
+      t.integer :aws_account_id
+      t.string :key
+      t.string :value
+      t.string :resource_type
+      t.integer :resource_id
+
+      t.timestamps
+    end
+  end
+
+  def self.down
+    drop_table :tags
+  end
+end

data/lib/aws_tracker/models/aws_account.rb

@@ -0,0 +1,69 @@
+require 'right_aws'
+
+module AWSTracker
+  class AWSAccount < ActiveRecord::Base
+
+    # Relationships
+    has_many :instances,        :dependent => :destroy
+    has_many :security_groups,  :dependent => :destroy
+    has_many :ebs_volumes,      {:dependent => :destroy, :class_name  => 'EBSVolume'}
+    has_many :tags,             :dependent => :destroy
+    # Validations
+    validates_presence_of :account_id, :access_key_id, :secret_access_key
+    validates_uniqueness_of :name, :account_id
+
+    # Configure logging
+    attr_accessor     :log
+    after_initialize  :setup_default_logger
+    def setup_default_logger
+      @log ||= AWSTracker.default_logger
+    end
+
+    def update_status
+      update_security_groups
+      update_instances
+      update_ebs_volumes
+      update_tags
+    end
+
+    # Fetches Security Group data and saves to database
+    def update_security_groups
+      SecurityGroup.refresh_from_account self
+    end
+
+    # Fetches Instance data and saves to database
+    def update_instances
+      Instance.refresh_from_account self
+    end
+
+    # Fetches EBS Volume data and saves to database
+    def update_ebs_volumes
+      EBSVolume.refresh_from_account self
+    end
+
+    # Fetches Tags and saves to database
+    def update_tags
+      Tag.refresh_from_account self
+    end
+
+    # Returns a right_aws connection to EC2, creating it on demand
+    attr_accessor :ec2  # Allow replacement of the connection for efficiency
+    def ec2
+      @ec2 ||= RightAws::Ec2.new(
+      access_key_id, secret_access_key, {:logger => @log}
+      )
+    end
+
+    # Creates and returns an account based on the environment variables:
+    # AWS_ACCOUNT_NUMBER, AWS_ACCESS_KEY_ID, AWS_ACCESS_KEY_SECRET
+    def self.from_memory
+      account_id = ENV['AWS_ACCOUNT_NUMBER']
+      AWSAccount.find_or_initialize_by_account_id(account_id) do |acc|
+        acc.name              = "M-R #{account_id}"
+        acc.access_key_id     = ENV['AWS_ACCESS_KEY_ID']
+        acc.secret_access_key = ENV['AWS_ACCESS_KEY_SECRET']
+      end
+    end
+    
+  end
+end

data/lib/aws_tracker/models/ebs_volume.rb

@@ -0,0 +1,55 @@
+module AWSTracker
+  class EBSVolume < ActiveRecord::Base
+
+    belongs_to :account,
+      :class_name  => 'AWSAccount',
+      :foreign_key => "aws_account_id"
+    belongs_to :instance
+    has_many :tags, :as => :resource, :dependent => :destroy
+
+    validates_presence_of :account, :volume_id, :size, :status
+    validates_uniqueness_of :volume_id
+
+    def self.refresh_from_account(acc)
+      acc.log.info "Updating EBS volumes for account '#{acc.name}'"
+      right_aws_array = acc.ec2.describe_volumes
+      right_aws_array.each do |right_aws_hash|
+        aws_id = right_aws_hash[:aws_id]
+        volume = EBSVolume.find_or_initialize_by_volume_id(aws_id) do |vol|
+          acc.ebs_volumes << vol
+          vol.account = acc
+        end
+        volume.update_from_right_hash(right_aws_hash)
+        if not volume.save
+          acc.log.error "Failed to save volume #{aws_id}: #{volume.errors}"
+        end
+      end
+      acc.log.info "Account '#{acc.name}' reports "+
+      "#{right_aws_array.count} EBS volumes"
+    end
+
+    def update_from_right_hash(right_hash)
+      self.volume_id          = right_hash[:aws_id]
+      self.size               = right_hash[:aws_size]
+      self.status             = right_hash[:aws_status]
+      self.availability_zone  = right_hash[:zone]
+      self.snapshot_id        = right_hash[:snapshot_id]
+      self.create_time        = right_hash[:aws_created_at]
+      self.attach_time        = right_hash[:aws_attached_at]
+      self.attachment_status  = right_hash[:aws_attachment_status]
+      self.device             = right_hash[:aws_device]
+      self.update_instance      right_hash[:aws_instance_id]
+    end
+
+    # Resets the instance attribute given an AWS instance ID
+    def update_instance(instance_id)
+      if instance_id == nil
+        self.instance = nil
+      else
+        self.instance = self.account.instances.find(:first,
+        :conditions => {:instance_id => instance_id})
+      end
+    end
+
+  end
+end

data/lib/aws_tracker/models/instance.rb

@@ -0,0 +1,71 @@
+module AWSTracker
+  class Instance < ActiveRecord::Base
+
+    belongs_to :account,
+      :class_name  => 'AWSAccount',
+      :foreign_key => "aws_account_id"
+    has_and_belongs_to_many :security_groups
+    has_many :ebs_volumes, :class_name  => 'EBSVolume'
+    has_many :tags, :as => :resource, :dependent => :destroy
+
+    validates_presence_of :account, :instance_id, :image_id, :instance_state
+    validates_uniqueness_of :instance_id
+
+    def self.refresh_from_account(acc)
+      acc.log.info "Updating instances for account '#{acc.name}'"
+      right_aws_array = acc.ec2.describe_instances
+      right_aws_array.each do |right_aws_hash|
+        aws_id = right_aws_hash[:aws_instance_id]
+        instance = Instance.find_or_initialize_by_instance_id(aws_id) do |inst|
+          acc.instances << inst
+          inst.account = acc
+        end
+        instance.update_from_right_hash(right_aws_hash)
+        instance.save
+      end
+      acc.log.info "Account '#{acc.name}' reports #{right_aws_array.count} instances"
+    end
+
+    def update_from_right_hash(rs_instance)
+      self.instance_id              = rs_instance[:aws_instance_id]
+      self.image_id                 = rs_instance[:aws_image_id]
+      self.instance_state           = rs_instance[:aws_state]
+      self.state_code               = rs_instance[:aws_state_code]
+      self.instance_type            = rs_instance[:aws_instance_type]
+      self.private_dns_name         = rs_instance[:private_dns_name]
+      self.dns_name                 = rs_instance[:dns_name]
+      self.key_name                 = rs_instance[:ssh_key_name]
+      self.ami_launch_index         = rs_instance[:ami_launch_index]
+      self.launch_time              = rs_instance[:aws_launch_time]
+      self.termination_time         = rs_instance[:aws_termination_time]
+      self.availability_zone        = rs_instance[:aws_availability_zone]
+      self.kernel_id                = rs_instance[:aws_kernel_id]
+      self.ramdisk_id               = rs_instance[:aws_ramdisk_id]
+      self.private_ip_address       = rs_instance[:private_ip_address]
+      self.ip_address               = rs_instance[:ip_address]
+      self.architecture             = rs_instance[:architecture]
+      self.root_device_type         = rs_instance[:root_device_type]
+      self.root_device_name         = rs_instance[:root_device_name]
+      self.virtualization_type      = rs_instance[:virtualization_type]
+      self.reason                   = rs_instance[:aws_reason]
+      self.state_reason_message     = rs_instance[:state_reason_message]
+      self.vpc_id                   = rs_instance[:vpc_id]
+      #:instance_life_cycle      = rs_instance[:XXXXXXXXX]
+      #:spot_instance_request_id = rs_instance[:XXXXXXXXX]
+      update_security_groups(rs_instance[:aws_groups])
+    end
+
+    # Updates this Instance's ActiveRecord relationship security_groups
+    # based on group_names, an array of strings
+    def update_security_groups(group_names)
+      new_groups = []
+      group_names.each do |name|
+        new_group = self.account.security_groups.find(:first,
+        :conditions => {:name => name})
+        new_groups.push(new_group) if new_group
+      end
+      self.security_groups = new_groups
+    end
+
+  end
+end

data/lib/aws_tracker/models/security_group.rb

@@ -0,0 +1,34 @@
+module AWSTracker
+  class SecurityGroup < ActiveRecord::Base
+
+    belongs_to :account,
+      :class_name  => 'AWSAccount',
+      :foreign_key => "aws_account_id"
+    has_and_belongs_to_many :instances
+
+    validates_presence_of :account, :name, :description, :owner
+    validates_uniqueness_of :name, :scope => "aws_account_id"
+
+    def self.refresh_from_account(acc)
+      acc.log.info "Updating security groups for account '#{acc.name}'"
+      right_aws_array = acc.ec2.describe_security_groups
+      right_aws_array.each do |rs_group|
+        group_to_update = (acc.security_groups.select() do |grp|
+          grp.name == rs_group[:aws_group_name]
+        end).first
+        group_to_update ||= SecurityGroup.new(:account => acc)
+        acc.security_groups << group_to_update
+        group_to_update.update_from_right_hash(rs_group)
+        group_to_update.save
+      end
+      acc.log.info "Account '#{acc.name}' reports #{right_aws_array.count} groups"
+    end
+
+    def update_from_right_hash(rs_group)
+      self.name        = rs_group[:aws_group_name]
+      self.description = rs_group[:aws_description]
+      self.owner       = rs_group[:aws_owner]
+    end
+
+  end
+end