cloud_financial_officer 0.1.0

data/.gitignore

@@ -0,0 +1,14 @@
+*.gem
+.bundle
+config/accounts.yml
+config/database.yml
+config/policies
+pkg/*
+*~
+.DS_Store
+*.tmproj
+db/*.sqlite3
+log/*.log
+tmp/**
+doc/**
+.yardoc

data/.powrc

@@ -0,0 +1 @@
+export RACK_ENV=development

data/.yardopts

@@ -0,0 +1 @@
+--no-private --protected lib/**/*.rb  - API.md

data/API.md

@@ -0,0 +1,146 @@
+Cloud Financial Officer REST API
+================
+The REST API exposes 3 resources:
+
+  1. Account - represents information about the accounts being tracked.
+  2. Resource - represents an cloud computing asset, like a Server or S3 Bucket.
+  3. Report - represents a group of billing records. Each billing record
+    represents a "line item" for a given Resource over a given time.
+
+
+----------------
+Account
+----------------
+Accounts are the "top-level" entity exposed by the API.
+Each Account has a list of its Resources.
+
+**RESOURCE LOCATION**: `/api/v1/accounts/[account-name]`
+
+**DISCOVERABLE AT**: `api/v1/accounts`
+             **OR**: `api/v1`
+
+**RESULT FIELDS**:
+
+  * id:             the name of the account
+  * service:        the name of the cloud service
+  * provider:       the name of the cloud service provider
+  * url:            link to an individual account
+  * report_url:     link to a report for an individual account
+  * resources:      an Array of links to an account's Resources
+  * billing_codes:  an Array of String pairs representing the billing codes
+                    for all the resources in this account
+
+
+----------------
+Resource
+----------------
+Resources represent individual cloud computing assets, and are always scoped
+within an Account.
+
+**RESOURCE LOCATION**: `/api/v1/accounts/[account-name]/[resource-type]/[resource-id]`
+
+**DISCOVERABLE AT**: `/api/v1/accounts/[account-name]`
+
+**RESULT FIELDS**:
+
+  * resource_id:    the cloud provider's ID value for this resource
+  * resource_type:  "Server", "Volume", "S3 Bucket", etc.
+  * account:        this resource's account information
+  * report_url:     link to a report for a specific resource
+  * billing_codes:  an Array of String pairs representing the billing codes
+                    for this resource
+
+
+----------------
+Report
+----------------
+Reports are a convenient way of summarizing costs for different combinations of
+accounts, billing codes, resource types, etc..
+In addition to the cost summary, each Report also exposes a list of billing
+records that make up the total cost of the report.
+Each billing record describes the costs incurred for one or more Resources
+over a given duration.
+
+**RESOURCE LOCATION**: `/api/v1/report/[for_[CONSTRAINT]]/[by_[GROUP]]`
+
+**DISCOVERABLE AT**: `/api/v1/accounts/[account-name]`
+             **OR**: `/api/v1/accounts/[account-name]/[resource-type]/[resource-id]`
+
+
+**CONSTRAINT EXPRESSIONS**: These limit the scope of the information in the report.
+  Constraints be chained together, irrespective of order.
+
+  * `for_account/[account1](/[account2]...)`
+  * `for_resource/[resourceID1](/[resourceID2]...)`
+  * `for_type/[resource_type1](/[resource_type2]...)`
+  * `for_provider/[provider1](/[provider2]...)`
+  * `for_service/[service1](/[service2]...)`
+  * `for_billing_type/[billing_type1](/[billing_type2]...)`
+  * `for_time/[start_datetime]/[stop_datetime]`
+  * `for_code/[key1]/[value1](/[key2]/[value2]...)`
+
+**GROUPING EXPRESSIONS**: These limit the number of records in the report, but
+  not the total cost or duration of the report itself. Each grouping expression
+  causes all records that share the same value for the given grouping attribute
+  to be combined into one record (much like a SQL GROUP_BY clause).
+  Grouping expressions can be chained together, irrespective of order.
+
+  * `by_account`
+  * `by_resource`
+  * `by_type`
+  * `by_provider`
+  * `by_service`
+  * `by_billing_type`
+
+**QUERY PARAMETERS**:
+
+  * content-type:  text/json or text/csv
+
+**REPORT SUMMARY FIELDS**:
+
+  * start_time:     start time for the earliest billing record in this report
+  * stop_time:      stop time for the latest billing record in this report
+  * total_cost:     cost, in cents, for all the billing records in this report
+  * cost_per_hour:  average hourly cost over all this report's records
+  * account:        this resource's account information
+  * billing_codes:  all the billing codes for all this report's records
+  * billing_records: Array of billing records that make up this report
+
+
+----------------
+Report Generation Details
+----------------
+
+Each Report represents a set of billing records. At the most granular level,
+each billing record represents a period of time during which a specific cloud
+resource incurred charges at a given hourly rate. And if the report was not
+generated with a `by_[GROUP]` expression in the URL, then each billing record
+in the report contains resource-specific data in all the following fields:
+
+**BILLING RECORD FIELDS**:
+
+  * service:        the name of the resource's cloud service
+  * provider:       the provider for the resource's cloud service
+  * account:        the name of the resource's account
+  * resource_id:    the cloud provider's ID value for this resource
+  * resource_type:  "Server", "Volume", "S3 Bucket", etc
+  * billing_type :  Some resources, like RDS servers, are billed for both runtime
+                    and storage. These costs are distinguished by billing_type.
+  * start_time:     start time for this billing record
+  * stop_time:      stop time for this billing record
+  * cost_per_hour:  hourly cost for this resource over this record's duration
+  * total_cost:     cost, in cents, for this resource over this record's duration
+
+However, since that level of detail is needed only infrequently, most useful
+reports group their records by one or more of the fields that make up each record.
+In this case, all records that share the same value for this grouping field are
+combined into a "composite record". The values for all the NON-grouping fields in
+a composite record are set to "*" (which stands for "more than one value") for any
+fields where the component records don't all share the same value.
+
+
+----------------
+Report Generation Examples
+----------------
+
+[TODO]

data/Gemfile

@@ -0,0 +1,2 @@
+source "http://rubygems.org"
+gemspec

data/Guardfile

@@ -0,0 +1,10 @@
+# A sample Guardfile
+# More info at https://github.com/guard/guard#readme
+
+guard 'rspec', 
+  :version => 2, 
+  :cli => "--color --format documentation -r ./spec/spec_helper.rb" do
+    watch(%r{^spec/.+_spec\.rb$})
+    watch(%r{^lib/(.+)\.rb$})     { |m| "spec/lib/#{m[1]}_spec.rb" }
+    watch('spec/spec_helper.rb')  { "spec" }
+end

data/README.md

@@ -0,0 +1,141 @@
+Cloud Financial Officer
+================
+Watches multiple cloud computing accounts, records expenses by resource,
+and exposes a REST API to query the recorded costs.
+
+
+----------------
+What is it?
+----------------
+Cloud Financial Officer is an application and library for tracking and
+reporting on the expenses generated by one or more cloud computing accounts.
+It monitors all the accounts and computes costs for each cloud computing
+"Resource", like a server (EC2 instance) or a storage container (S3 bucket).
+It saves the resulting billing records into a database using ActiveRecord, and
+generates JSON reports on this data.
+
+The cloud computing Resources can be assigned "billing codes", which are
+arbitrary pairs of Strings that can be used by an organization for internal
+billing. Each Resource's billing codes are attached to all its billing records.
+
+By default, a useful set of Resource coding policies is included. For example,
+any Tags that are assigned to EC2 Resources are automatically converted to
+billing codes. Billing codes from Security Groups are passed on to the Instances
+that run within them, and from there to the EBS volumes attached to the
+Instances. Custom policies can also be written (in Ruby) to assign billing codes
+based on arbitrary logic.
+
+
+----------------
+Why is it?
+----------------
+The principal motivation behind this software is to provide "microbilling" for
+Amazon Web Services, as explained in the README for the [Cloud Cost Tracker].
+
+Though every effort is made to calculate costs accurately, the purpose of CFO
+is not really to produce an exact number matching the bill from a given service
+provider. It is more useful in determining the relative breakdown of costs
+by provider, account, or resource type (which it does automatically), and more
+useful still in reporting costs for internal billing units, like customer or
+project. This can be achieved with EC2 tags or custom resource coding logic.
+
+
+----------------
+Installation
+----------------
+CFO can be used as a standalone application or as a Rack middleware library.
+
+###   To install as an application  ###
+
+  1) Install project dependencies:
+
+        gem install rake bundler
+
+  2) Fetch the code:
+
+        git clone https://github.com/benton/cloud_financial_officer.git
+        cd cloud_financial_officer
+
+  3) Now edit the `Gemfile` to include your desired database driver.
+     Then bundle everything together.
+
+        bundle
+
+###   To install as a library   ###
+
+  1) Install the gem, or add it as a Bundler dependency and `bundle`.
+
+        gem install cloud_financial_officer
+
+  2) Require the middleware from your Rack application, then insert it
+    in the stack:
+
+        require 'cloud_financial_officer'
+        ...
+        config.middleware.use CFO::Service  # (Rails application.rb)
+                                            # or
+        use CFO::Service                    # (Sinatra)
+
+  3) Require the database migration tasks in your `Rakefile`:
+
+        require 'cloud_cost_tracker/tasks'
+
+
+###   Initializing the database   ###
+
+  1) Create a Rails-style `config/database.yml` file (example included).
+
+  2) Create the tables. *`ENV['RACK_ENV']` determines the configuration entry*
+
+        rake db:migrate:tracker
+
+
+----------------
+Usage
+----------------
+CFO consists of two parts:
+
+* A rake task that gathers cost information.
+* A web service that exposes a REST API for the billing records.
+
+Both require information about which accounts to track and report on.
+This is stored in `config/accounts.yml` (see the provided example).
+Both also require a `config/database.yml`.
+
+The tracking process is run with:
+
+      rake track
+
+The service is run with:
+
+      rake service
+or
+
+      rackup
+
+The entry point for the REST API is `/api/v1/accounts`
+(See the {file:API.md API documentation})
+
+
+----------------
+Development
+----------------
+
+*Getting started with development*
+
+1) Install project dependencies
+
+    gem install rake bundler
+
+2) Fetch the project code
+
+    git clone https://github.com/benton/cloud_financial_officer.git
+    cd cloud_financial_officer
+
+3) Bundle up and run the tests
+
+    bundle && rake
+
+
+
+[Cloud Cost Tracker]: https://github.com/benton/cloud_cost_tracker

data/Rakefile

@@ -0,0 +1,20 @@
+# Set up bundler
+%w{rubygems bundler bundler/gem_tasks}.each {|dep| require dep}
+bundles = [:default]
+Bundler.setup(:default)
+require 'cloud_financial_officer'
+case CFO.env
+when 'development'  then Bundler.setup(:default, :development)
+when 'test'         then Bundler.setup(:default, :development, :test)
+end
+
+CFO.force_timezone  # This does nothing by default - modify it if you need it
+
+# Load all tasks from the cloud_cost_tracker gem
+require 'cloud_cost_tracker/tasks'
+
+# Load all tasks from 'lib/tasks'
+Dir["#{File.dirname(__FILE__)}/lib/tasks/*.rake"].sort.each {|ext| load ext}
+
+desc 'Default: runs all tests'
+task :default => :spec

data/cloud_financial_officer.gemspec

@@ -0,0 +1,43 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "cfo/version"
+
+Gem::Specification.new do |s|
+  s.name        = "cloud_financial_officer"
+  s.version     = CFO::VERSION
+  s.authors     = ["Benton Roberts"]
+  s.email       = ["benton@bentonroberts.com"]
+  s.homepage    = "http://github.com/benton/cloud_financial_officer"
+  s.summary     = %q{Watches multiple cloud computing accounts, }+
+                  %q{records expenses by resource, and exposes }+
+                  %q{a REST API to query the recorded costs}
+  s.description = %q{Watches multiple cloud computing accounts, }+
+                  %q{records expenses by resource, and exposes }+
+                  %q{a REST API to query the recorded costs}
+  s.rubyforge_project = "cloud_financial_officer"
+
+  # This project is both a Gem and an Application,
+  # so the Gemfile.lock is included in the repo for application users,
+  # but excluded from the packaged Gem, for middleware use.
+  git_files       = `git ls-files`.split("\n")    # Read all files in the repo,
+  git_files.delete "Gemfile.lock"                 # remove Gemfile.lock, and
+  s.files         = git_files                     # use the result in the Gem.
+
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  # Runtime dependencies
+  s.add_dependency "sinatra"
+  s.add_dependency "sinatra-contrib"
+  s.add_dependency "cloud_cost_tracker", ">=0.1.0"
+
+  # Development / Test dependencies
+  s.add_development_dependency "rake"
+  s.add_development_dependency "rspec"
+  s.add_development_dependency "factory_girl", "~> 2.1.0"
+  s.add_development_dependency "guard"
+  s.add_development_dependency "guard-rspec"
+  s.add_development_dependency "ruby_gntp"
+  s.add_development_dependency "sqlite3"
+end

data/config/accounts.example.yml

@@ -0,0 +1,45 @@
+AWS EC2 production account: 
+  :provider: AWS
+  :service: Compute
+  :credentials: 
+    :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
+    :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+  :delay: 120
+  :exclude_resources:
+AWS S3 account:
+  :provider: AWS
+  :service: Storage
+  :credentials:
+    :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
+    :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+  :delay: 120
+  :exclude_resources:
+#  - :directories # (buckets)
+  - :files # secondary entity - does not work yet
+AWS RDS account:
+  :provider: AWS
+  :service: RDS
+  :credentials:
+    :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
+    :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+  :delay: 60
+  :exclude_resources:
+  - :parameters # secondary entity - does not work yet
+AWS Route 53 account:
+  :provider: AWS
+  :service: DNS
+  :credentials:
+    :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
+    :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+  :delay: 60
+  :exclude_resources:
+  - :records # secondary entity - does not work yet
+AWS Elasticache account:
+  :provider: AWS
+  :service: Elasticache
+  :credentials:
+    :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
+    :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+  :delay: 60
+  :exclude_resources:
+

data/config/database.example.yml

@@ -0,0 +1,18 @@
+development:
+  username: root
+  pool: 10             # Best to use at lease one connection per tracked account
+  timeout: 10000       # Leave this nice and long
+  adapter: sqlite3
+  database: db/cfo_development.sqlite3
+test:
+  username: root
+  pool: 10             # Best to use at lease one connection per tracked account
+  timeout: 10000       # Leave this nice and long
+  adapter: sqlite3
+  database: db/cfo_test.sqlite3
+production:
+  username: root
+  pool: 10             # Best to use at lease one connection per tracked account
+  timeout: 10000       # Leave this nice and long
+  adapter: sqlite3
+  database: db/cfo_production.sqlite3

data/config.ru

@@ -0,0 +1,2 @@
+require './lib/cfo/service'
+run CFO::Service

data/lib/cfo/cfo.rb

@@ -0,0 +1,44 @@
+require 'cloud_cost_tracker'
+module CFO
+  require File.join(File.dirname(__FILE__), 'resources/account')
+
+  # Returns the current RACK_ENV, or 'development' if not set
+  # @return [String] ENV[RACK_ENV] || ENV[RAILS_ENV] || development
+  def self.env ; ::CloudCostTracker.env end
+
+  # Connects to the database defined in db_file.
+  # Uses the YAML section indexed by CloudCostTracker#env.
+  # @param db_file the path to a Rails-style YAML DB config file.
+  # @return [Hash] the current envinronment's database configuration.
+  def self.connect_to_database(db_file = ENV['DB_CONFIG_FILE'])
+    ::CloudCostTracker.connect_to_database(db_file)
+  end
+
+  # Loads account information defined in account_file.
+  # @param account_file the path to a YAML file (see accounts.yml.example).
+  # @return [Array<Account>] an Array of Account resources.
+  def self.read_accounts(account_file = ENV['ACCOUNT_FILE'])
+    account_hash = ::CloudCostTracker.read_accounts(account_file)
+    account_hash.map do |(account_name, account_info)|
+      account = Account.new(account_info.merge(:name => account_name))
+    end
+  end
+
+  PROJECT_DIR = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
+  require "#{PROJECT_DIR}/lib/cfo/version"
+  # Defines the root URI path of the CFO service.
+  # @return [String] the root URI path of the CFO service
+  def self.root
+    "/api/v#{CFO::API_VERSION}"
+  end
+
+  # Forces the ActiveRecord timezone settings,
+  # which normally defaults to :local.
+  # Uncomment the code in this function ONLY if you're using this CFO as a
+  # standalone application, and need to force the time zone.
+  # See: http://api.rubyonrails.org/classes/ActiveRecord/Timestamp.html
+  def self.force_timezone
+    #ActiveRecord::Base.default_timezone = :utc
+  end
+
+end

data/lib/cfo/record_filter.rb

@@ -0,0 +1,80 @@
+module CFO
+  # Encapsulates sevral algorithms for for changing one set of BillingRecords
+  # into another.
+  module RecordFilter
+    include CloudCostTracker::Billing
+
+    # Truncates each record in records so that its start time >= start_time
+    # and its stop time <= stop_time.
+    # @param [Array<BillingRecord>] records the records to filter
+    # @param [DateTime] start_time the earliest time allowed in the records
+    # @param [DateTime] stop_time the latest time allowed in the records
+    # @return [Array<BillingRecord>] the modified records
+    def self.truncate_by_time(records, start_time, stop_time)
+      records.each do |record|
+        record.start_time = start_time if record.start_time < start_time
+        record.stop_time  = stop_time  if record.stop_time  > stop_time
+        duration = Time.parse(record.stop_time.to_s) -
+                   Time.parse(record.start_time.to_s)
+        record.total_cost = duration * record.cost_per_hour / SECONDS_PER_HOUR
+      end
+      records
+    end
+
+    # Filters an Array of BillingRecords by start and stop times.
+    # @param [Array<BillingRecord>] records the records to filter
+    # @param [Array<Hash>] constraints an Array of Hash constraints on the
+    #   BillingRecords to be retrieved. See {#CFO::Record#initialize}.
+    #   This method acts only on the 'for_time' constraint key, if it exists.
+    # @return [Array<BillingRecord>] records that match the time constraints
+    #   in @constraint_array
+    def self.by_time(records, constraints)
+      time_constraint = constraints.find {|c| c['for_time']}
+      return records if time_constraint == nil
+      time_strings = time_constraint['for_time']
+      return records if time_strings.empty?
+      times = (time_strings.map do |time| # Convert values to DateTime
+        time.is_a?(Time) ? time : Time.parse(time.to_s)
+      end).sort
+      times << Time.now if times.size.odd?
+      results = records.select do |record|
+        (record.start_time < times.last) and
+        (record.stop_time > times.first)
+      end
+      RecordFilter.truncate_by_time(results, times.first, times.last)
+    end
+
+    # Filters an Array of BillingRecords by BillingCode keys and values
+    # @param [Array<BillingRecord>] records the records to filter
+    # @param [Array<Hash>] constraints an Array of Hash constraints on the
+    #   BillingRecords to be retrieved. See {#CFO::Record#initialize}.
+    #   This method acts only on the 'for_code' constraint key, if it exists.
+    # @return [Array<BillingRecord>] records that match the code constraints
+    #   in @constraint_array
+    def self.by_code(records, constraints)
+      code_constraint = constraints.find {|c| c['for_code']}
+      return records if code_constraint == nil
+      code_strings = code_constraint['for_code']
+      return records if code_strings.empty?
+      # Make pairs from the array of code_strings
+      code_strings << '*' if code_strings.size.odd?
+      code_pairs = (0..(code_strings.size/2)-1).map do |index|
+        [code_strings[index*2], code_strings[(index*2)+1]]
+      end
+      matching_records = Array.new
+      records.each do |record|
+        record.billing_codes.map do |code|
+          code_pairs.map do |code_pair|
+            if ((code_pair[0] == '*')      and (code_pair[1] == code.value)) ||
+               ((code_pair[0] == code.key) and (code_pair[1] == '*'))        ||
+               ((code_pair[0] == code.key) and (code_pair[1] == code.value))
+                  matching_records << record
+            end
+          end
+        end
+      end
+      matching_records
+    end
+
+  end
+end