dash-bees 0.18

data/CHANGELOG

@@ -0,0 +1,116 @@
+2010-09-16 v0.18 Now bees, have separate update and webhook methods
+
+Update method separated from webhook method.
+
+Bees for being busy workers.
+
+2010-09-12 v0.17 API change to callback and HTTP access
+
+Changed API to use callback object instead of block with too many argument
+combinations.
+
+Clarified and validating that source state can only use alphanumeric/underscore
+in field names.
+
+Added wrapper around HTTP API to make it easier to switch to asynchronous
+processing later on.
+
+2010-09-08 v0.16 To name a source, set source.name
+
+During setup, name the source by setting source.name, not metric.name.
+
+Revised and improved test suite.
+
+2010-09-07 v0.15 Renamed to DashFu::Mario
+
+2010-09-07 v0.14 RubyGems and Github identities
+
+RubyGems releases now identify RubyGems as the source of the activity.
+
+Github source includes github login name as identity.
+
+2010-08-31 v0.13 Github data source merges related commits
+
+Now it does.
+
+2010-08-31 v0.12 Github data source merges related commits
+
+Github data source merges related commits (same committer/time) into single
+activity and shows first line of all commit messages.
+
+Changed to coding/testing with Ruby 1.9.2, since this is the target
+environment.
+
+2010-08-30 v0.11 Changes to activity person, AS 3.0 support
+
+Activity's person name is now fullname, photo is photo_url and url is no
+longer, but we do accept multiple identities of the form domain:id, e.g
+twitter.com:assaf, linkedin.com:assafarkin.
+
+Test suite now allows validating metric, activity and person. Call the validate
+method to run validation and raise exception on error (with a helpful error
+message). Call valid? if you're only interested in true/false.
+
+Gem dependencies removed to allow running with ActiveSupport 3.0.
+
+2010-08-26 v0.10 Tags and better messages
+
+Activities can now include tags.
+
+Improvement all around to all the HTML messages.
+
+2010-08-25 v0.9 Better Github commit message
+
+2010-08-24 v0.8 Minor API change
+
+Attribute :id is now :uid.
+
+Activity content can be set using either :html or :text attribute.
+2010-08-24 v0.7 Added test suite and resources
+
+Each source can have an associated resources file (YAML), e.g. ruby_gems.rb
+would have ruby_gems.yml. The default implementation of methods like name,
+description and display pull content from the resource. Resources can support
+multiple languages, but we're starting with just EN.
+
+Some sources also need API keys. These are accessed using the method api_key
+that returns the API key value for the current source (often a string, but can
+also be hash or array). There's an api_keys.yml file which contains fake API
+keys. You can put a real key in there while generating a cassette for your test
+case.
+
+Test suite is here, using WebMock and VCR to rest API calls. Cassettes go in
+test/cassettes, fixtures in test/fixtures.
+
+2010-08-19 v0.6 Added Backtweet
+
+2010-08-19 v0.5 Added Github and Github issues
+
+2010-08-18 v0.4 Metrics that set and increment
+
+Revised setup method to change special values metric.name and metric.columns
+(was name and column).
+
+There are two types of metrics, those that collect totals and those that
+collect daily/hourly values. The setup method indicates that by setting the
+value metric.total (defaults to false).
+
+Revised update method to allow either setting current value (:set) or
+incrementing current value (:inc).
+
+2010-08-11 v0.3 Always be sending changes
+
+Changed: Collector update methods accepts inc and timestamp arguments (but set
+is gone).
+
+Fixed: Rubygems source captures most recent downloads count as meta-data,
+updates collector with change since last update.
+
+Fixed: Rubygems source can handle whatever name you throw at it and properly
+escapes it.
+
+2010-08-10 v0.2 Working Rubygems source
+
+Fixed: Rubygems source not updating meta data like authors, project info, URL.
+
+2010-08-09 v0.1 First release

data/Gemfile

@@ -0,0 +1,16 @@
+source "http://rubygems.org/"
+gemspec
+
+group :development do
+  gem "rake"
+  gem "yard"
+end
+
+group :test do
+  gem "awesome_print"
+  gem "nokogiri"
+  gem "shoulda"
+  gem "timecop"
+  gem "vcr"
+  gem "webmock"
+end

data/MIT-LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2010 Assaf Arkin
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.rdoc

@@ -0,0 +1,113 @@
+{Dash-fu}[http://dash-fu.com] sources need to get their data somehow: enter
+Bees. Bees are the brave workers that go out and collect data from various
+data sources and report them back in the form of activity stream, metrics and
+status updates.
+
+
+== Setup
+
+Dash-fu creates data sources using a four-step process. In the first step, it
+calls the display method to get HTML fragments and display these to the user.
+
+Currently it uses two fragments:
+
+* inputs -- HTML input controls for setting up a new controller.
+* notes -- Additional setup notes: other steps that need to be
+  followed, what values to supply, etc.
+
+Input controls use names of the form source[..param..], and wrapped inside label
+elements. For example:
+
+  <label>Screen name <input type="text" name="source[screen_name]" size="30"></label>
+
+Once the user fills and submits the form, Dash-fu creates a new empty state
+object and calls the setup method with that object and the input fields.
+
+The state object is a Hash than can store basic values: nil, string, boolean,
+numerics, arrays and other hashes. Key names must be alphanumeric/underscore.
+Bees use it to store state in between method calls.
+
+The setup method must set one value, "source.name", to suggest a name for this
+source, e.g. "Twitter mentions for @dash_fu".
+
+In the third state, Dash-fu calls the validate method. If this method raises an
+exceptions, the error message is shown to the user and the source discarded. The
+fourth step involves registering callbacks (not currently supported).
+
+
+== Metrics
+
+For sources that include metrics, the setup method specifies the follow values
+during the call to setup:
+
+* metric.columns -- Columns (at least one) as an array of hashes, see
+  attributes names below.
+* metric.totals -- True if the metric collects life-time totals (e.g.
+  downloads, uptime), false if it only collects recent values (e.g. average
+  response time).
+
+Columns are specified using the following attributes:
+
+* id -- Column identifier (if missing, derived from column name)
+* name -- Column name (required)
+
+
+== Updates
+
+Periodically, all sources are updated by calling their update method. This
+method taks two arguments, the source state object and an object wrapping
+several callback methods.
+
+A single update can call any combination of callback methods, for example, it
+can set values in a metric and record an activity.
+
+The callback methods are:
+
+* set! -- Sets the most recent value for this metric. The single argument is a
+  hash, where key names are either column ids or indices, and values are
+  integers or floats.
+* inc! -- Changes the most recent value for this metric. The single argument is
+  a hash, where key names are either column ids or indices, and values are
+  integers or floats.
+* activity! -- Adds an activity to the stream, see below for details about the
+  argument.
+* error! -- Records a processing error. The single argument is an error
+  message. Since updates are processed asynchronously, use this method to
+  indicate any processing error, don't raise an exception.
+
+Activities have the following attributes:
+
+* uid -- Unique identifier (within the scope of this source). Optional.
+* html -- HTML contents of the activity.
+* text -- Text contents of the activity (alternative to html attribute).
+* url -- Points back to the original resource. Optional.
+* tags -- Any number of tags for identifying related activities. Tags are
+  lower-cased and can contain alphanumeric and dashes. Optional.
+* person -- Person who performed this activity (see below).
+* timestamp -- Timestamp activity occurred. Optional.
+
+HTML can use links (HTTP/S and email), images, bold, italics, paragraphs, block
+quotes, lists, tables, pre-formatted text and even video. Scripts, objects and
+frames are stripped out, as are any script and style attributes.
+
+People are identified by the following attributes:
+
+* fullname -- What it says on the label.
+* email -- Email address if known.
+* identities -- Any number of identities, in the form domain:id, e.g.
+  twitter.com:assaf or linkedin.com:assafarkin.
+* photo_url -- Preferrably 48x48.
+
+In addition, from time to time, Dash-fu would call the meta method and display
+the returned fields. This method returns an array of hashes, each describing a
+single display value using the following attributes.
+
+* title -- Title to show next to the value (optional).
+* text -- Text to show as value of this field.
+* url -- Link (optional).
+
+
+== Webhooks
+
+TBD
+

data/Rakefile

@@ -0,0 +1,39 @@
+require "rake/testtask"
+
+# -- Building stuff --
+
+spec = Gem::Specification.load(Dir["*.gemspec"].first)
+
+desc "Build the Gem"
+task :build do
+  sh "gem build #{spec.name}.gemspec"
+end
+
+desc "Install #{spec.name} locally"
+task :install=>:build do
+  sudo = "sudo" unless File.writable?( Gem::ConfigMap[:bindir])
+  sh "#{sudo} gem install #{spec.name}-#{spec.version}.gem"
+end
+
+desc "Push new release to gemcutter and git tag"
+task :push=>["test", "build"] do
+  sh "git push"
+  puts "Tagging version #{spec.version} .."
+  sh "git tag v#{spec.version}"
+  sh "git push --tag"
+  puts "Building and pushing gem .."
+  sh "gem push #{spec.name}-#{spec.version}.gem"
+end
+
+
+task :default=>:test
+Rake::TestTask.new do |task|
+  task.test_files = FileList['test/**/*_test.rb']
+  #task.warning = true
+  task.verbose = true
+end
+
+task :clobber do
+  rm_rf "doc"
+  rm "test/test.log"
+end

data/dash-bees.gemspec

@@ -0,0 +1,25 @@
+$: << File.dirname(__FILE__) + "/lib"
+require "dash-fu/bee"
+
+Gem::Specification.new do |spec|
+  spec.name           = "dash-bees"
+  spec.version        = DashFu::Bee::VERSION
+  spec.author         = "Assaf Arkin"
+  spec.email          = "assaf@labnotes.org"
+  spec.homepage       = "http://dash-fu.com"
+  spec.summary        = "Hopping around collecting data, keeping your dashes dashing"
+
+  spec.files          = Dir["{bin,lib,test}/**/*", "CHANGELOG", "MIT-LICENSE", "README.rdoc", "Rakefile", "Gemfile", "*.gemspec"]
+
+  spec.has_rdoc         = true
+  spec.extra_rdoc_files = "README.rdoc", "CHANGELOG"
+  spec.rdoc_options     = "--title", "DashFu::Bee #{spec.version}", "--main", "README.rdoc",
+                          "--webcvs", "http://github.com/assaf/#{spec.name}"
+
+  spec.required_ruby_version = '>= 1.8.7'
+  spec.add_dependency "activesupport"
+  spec.add_dependency "json"
+  spec.add_dependency "nokogiri"
+  spec.add_dependency "rack"
+  spec.add_dependency "i18n"
+end

data/lib/dash-fu/bee.rb

@@ -0,0 +1,212 @@
+require "active_support"
+require "json"
+require "net/http"
+require "rack"
+require "uri"
+
+# See http://dash-fu.com
+module DashFu
+
+  # The README covers it all.
+  module Bee
+
+    VERSION = "0.18"
+
+    class << self
+      attr_accessor :logger
+
+      # Returns all available bees.
+      def all
+        @bees ||= {}
+      end
+
+      # Returns bee by its identifier.
+      def find(id)
+        all[id]
+      end
+
+      # Loads all the bees from the given directory. The bee identifier is
+      # derived from the filename (e.g. all/get_this.rb becomes "get_this"). The
+      # bee class must map to the source identifier within the Bee module, e.g.
+      # DashFu::Bee::GetThis).
+      def load_bees(path = File.dirname(__FILE__) + "/bees")
+        Dir["#{path}/*.rb"].each do |file|
+          id = File.basename(file, ".rb")
+          fail "Bee #{id} already loaded" if all[id]
+          load file
+          klass = Bee.const_get(id.camelize)
+          all[id] = klass.new
+          logger.info "Loaded Bee #{id}: #{klass}"
+        end
+      end
+
+      # API keys (see instance method api_key).
+      attr_accessor :api_keys
+
+      def included(klass)
+        klass.extend ClassMethods
+      end
+    end
+
+
+    module ClassMethods
+      # Bee identifier.
+      def bee_id
+        @bee_id ||= name.demodulize.underscore
+      end
+    end
+
+
+    # Returns the display name for this Bee.
+    #
+    # Uses the resource 'name', and fallbacks on the class name (e.g
+    # Bee::OneUp becomes "One Up").
+    def name
+      resources["name"] || bee_id.titleize
+    end
+
+    # Returns additional information about this source.
+    #
+    # A good description helps the user identity source and decide whether or
+    # not to use it.
+    #
+    # Uses the resource 'description'.
+    def description
+      resources["description"]
+    end
+
+    # This method returns a hash with two values:
+    # * inputs -- HTML fragment for a setup form
+    # * notes -- HTML fragment for setup notes
+    #
+    # Uses the resources 'inputs' and 'notes'.
+    def display
+      { :inputs=>resources["inputs"], :notes=>resources["notes"] }
+    end
+
+    # Called to setup a new source with parameters from the HTML form (see
+    # #display). If there are any missing values, report them when #validate is
+    # called.
+    def setup(source, params)
+    end
+
+    # Called to validate the source. If there are any error, raise an
+    # exception. A good error message will help the user understand which value
+    # is missing or invalid and how to correct that.
+    def validate(source)
+    end
+
+    # Called to register a Webhook (for sources that need it)
+    def register(source, url)
+    end
+
+    # Called to update the source. It can invoke callback methods any number of
+    # times with any combination of the supported named arguments for updating
+    # the source.
+    def update(source, callbacks)
+    end
+
+    # Called in response to (previously registered) Webhook request. Second
+    # argument is a Rack::Request. It can invoke callback methods any number of
+    # times with any combination of the supported named arguments for updating
+    # the source.
+    def webhook(source, request, callback)
+    end
+
+    # Called to unregister a Webhook (for sources that don't need it).
+    def unregister(source, url)
+    end
+
+    # Returns meta-data to be displayed alongside any other data. This method
+    # should return an array of hashes, each with the keys title (optional),
+    # text and url (optional). Good meta-data provides timely and relevant
+    # information that is not available in the raw data.
+    def meta(source)
+      []
+    end
+
+    # Use this to make HTTP request to external services. This method opens a
+    # new session and yields to the block. The block can them make HTTP requests
+    # on the remote host. The block may be called asynchronously.
+    def session(host, port = 80)
+      http = Net::HTTP.new(host, port)
+      http.use_ssl = true if port == 443
+      http.start do |http|
+        yield Session.new(http)
+      end
+    end
+
+    # HTTP Session.
+    class Session
+      def initialize(http) #:nodoc:
+        @http = http
+      end
+
+      # Make a GET request and yield response to the block. Response consists of
+      # three arguments: status code, response body and response headers. The
+      # block may be called asynchronoulsy.
+      def get(path, headers = {}, &block)
+        response = @http.request(get_request(path, headers))
+        yield response.code, response.body, {}
+      end
+
+      # Make a GET request and yield response to the block. If the response
+      # status is 200 the second argument is the response JSON object. The block
+      # may be called asynchronously.
+      def get_json(path, headers = {}, &block)
+        response = @http.request(get_request(path, headers))
+        if Net::HTTPOK === response
+          json = JSON.parse(response.body) rescue nil
+          if json
+            yield response.code.to_i, json, {}
+          else
+            yield 500, "Not a JSON document", {}
+          end
+        else
+          yield response.code.to_i, response.message, {}
+        end
+      end
+
+    protected
+
+      def get_request(path, headers)
+        request = Net::HTTP::Get.new(path)
+        request.basic_auth headers[:username], headers[:password] if headers[:username] && headers[:password]
+        request
+      end
+    end
+
+  protected
+
+    # Source identifier.
+    def bee_id
+      self.class.bee_id
+    end
+
+    # Logger. Dump messages that can help with troubleshooting.
+    def logger
+      DashFu::Bee.logger
+    end
+
+    # Returns I18n resources for this gem.
+    def resources
+      unless @resources
+        file_name = File.dirname(__FILE__) + "/bees/#{bee_id}.yml"
+        @resources = File.exist?(file_name) ? YAML.load_file(file_name)["en"] : {}
+      end
+      @resources
+    end
+
+    # Shortcut for Rack::Utils.escape_html
+    def escape_html(text)
+      Rack::Utils.escape_html(text.to_s)
+    end
+    alias :h :escape_html
+
+    # Returns API key for this source. May return string or hash, depending on
+    # the API.
+    def api_key
+      @api_key ||= DashFu::Bee.api_keys[bee_id] or raise "No API key for #{bee_id}"
+    end
+  end
+end