dash-mario 0.16 → 0.17

data/CHANGELOG

@@ -1,3 +1,14 @@
+2010-09-12 v0.17 API change to callback and HTTP access
+
+Changed API to use callback object instead of block with too many argument
+combinations.
+
+Clarified and validating that source state can only use alphanumeric/underscore
+in field names.
+
+Added wrapper around HTTP API to make it easier to switch to asynchronous
+processing later on.
+
 2010-09-08 v0.16 To name a source, set source.name
 
 During setup, name the source by setting source.name, not metric.name.

data/README.rdoc

@@ -41,6 +41,11 @@ For metric.columns, each item can use the following keys:
 * id -- Column identifier (if missing, derived from column name)
 * name -- Column name (required)
 
+Note: any state information you want to store can only use field names with
+alphanumeric charactercs and underscore. Periods and all other special
+characters are reserved for special field names that are not stored as part of
+the state.
+
 Shortly after setup, the validate method is called with the same context.  If it
 raises any exception, the error is displayed to the user and the source is
 discarded. Otherwise, register is called with a Webhook URL.  Some Marios use
@@ -50,25 +55,27 @@ that to register the Webhook with another service.
 == Updates
 
 Periodically, the Mario will be asked to update the source by calling the update
-method. This method is called with the same context and a block.  If data comes
-from a webhook, then the second argument to update is a Rack::Request object.
+method. This method is called with the same context and callbacks.  If data
+comes from a webhook, then the second argument to update is a Rack::Request
+object.
 
-The block passed to update can be used to update the source by yielding to it
-with a hash of named arguments. A single update may yield any number of times
-with any combination of arguments.
+The callback object passed to update can be used to update the source by calling
+one of its many methods.  A single update may call any combination of methods,
+e.g. increasing a metric and recording an activity.
 
-The named arguments are:
+The callback methods exposed by the callback object are:
 
-* set -- Columns to set (metric). Records the most recent value for this
-  metric. This is a hash where the keys are column ids (or indexes), the values
-  are integers or floats.
-* inc -- Columns to increment (metric). Records a change in value which may
-  be positive or negative. This is a hash where the keys are column ids (or
+* set! -- Columns to set (metric). Records the most recent value for this
+  metric. The single argument is a hash where the keys are column ids (or
   indexes), the values are integers or floats.
-* timestamp -- The Time (if missing, uses the current system time).
-* activity -- Activity to show in the stream. See below.
-
-Activity can specify the following attributes:
+* inc! -- Columns to increment (metric). Records a change in value which may
+  be positive or negative. The single argument is a hash where the keys are
+  column ids (or indexes), the values are integers or floats.
+* activity! -- Records a single activity. The single argument is a hash with
+  various values described below.
+* error! -- Records a processing error. The single agrument is an error message.
+
+An activity is specified using the following fields:
 
 * uid -- Unique identifier (within the scope of this source). For example,
   if activity is a release, this could be the version number.

data/dash-mario.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |spec|
   spec.name           = "dash-mario"
-  spec.version        = "0.16"
+  spec.version        = "0.17"
   spec.author         = "Assaf Arkin"
   spec.email          = "assaf@labnotes.org"
   spec.homepage       = "http://dash-fu.com"

data/lib/dash-fu/mario.rb

@@ -99,11 +99,11 @@ module DashFu
     end
 
     # Called to update the source. This method will be called periodically with
-    # a source and a block. When triggered by a Webhook, it will be called with
-    # source, Rack::Request and a block. It can yield to the block any number of
-    # times with any combination of the supported named arguments for updating
-    # the source.
-    def update(source, request, &block)
+    # a source and a callback. When triggered by a Webhook, it will be called
+    # with source, Rack::Request and a block. It can invoke callback methods any
+    # number of times with any combination of the supported named arguments for
+    # updating the source.
+    def update(source, request, callbacks)
     end
 
     # Called to unregister a Webhook (for sources that don't need it).
@@ -118,6 +118,57 @@ module DashFu
       []
     end
 
+    # Use this to make HTTP request to external services. This method opens a
+    # new session and yields to the block. The block can them make HTTP requests
+    # on the remote host. The block may be called asynchronously.
+    def session(host, port = 80)
+      http = Net::HTTP.new(host, port)
+      http.use_ssl = true if port == 443
+      http.start do |http|
+        yield Session.new(http)
+      end
+    end
+
+    # HTTP Session.
+    class Session
+      def initialize(http) #:nodoc:
+        @http = http
+      end
+
+      # Make a GET request and yield response to the block. Response consists of
+      # three arguments: status code, response body and response headers. The
+      # block may be called asynchronoulsy.
+      def get(path, headers = {}, &block)
+        response = @http.request(get_request(path, headers))
+        yield response.code, response.body, {}
+      end
+
+      # Make a GET request and yield response to the block. If the response
+      # status is 200 the second argument is the response JSON object. The block
+      # may be called asynchronously.
+      def get_json(path, headers = {}, &block)
+        response = @http.request(get_request(path, headers))
+        if Net::HTTPOK === response
+          json = JSON.parse(response.body) rescue nil
+          if json
+            yield response.code.to_i, json, {}
+          else
+            yield 500, "Not a JSON document", {}
+          end
+        else
+          yield response.code.to_i, response.message, {}
+        end
+      end
+
+    protected
+
+      def get_request(path, headers)
+        request = Net::HTTP::Get.new(path)
+        request.basic_auth headers[:username], headers[:password] if headers[:username] && headers[:password]
+        request
+      end
+    end
+
   protected
 
     # Source identifier.

data/lib/dash-fu/marios/backtweets.rb

@@ -6,7 +6,7 @@ module DashFu::Mario
     def setup(source, params)
       url = params["url"].strip.downcase.sub(/^http(s?):\/\//, "")
       source["source.name"] = "Tweets for #{url}"
-      source["metric.columns"] = [{ :id=>"tweets", :label=>"Tweets" }]
+      source["metric.columns"] = [{ id: "tweets", label: "Tweets" }]
       source["metric.totals"] = true
       source["url"] = url
     end
@@ -15,43 +15,35 @@ module DashFu::Mario
       raise "Missing URL" if source["url"].blank?
     end
 
-    def update(source, request, &block)
-      Net::HTTP.start "backtweets.com" do |http|
-        get = Net::HTTP::Get.new("/search.json?key=#{api_key}&q=#{Rack::Utils.escape source["url"]}")
-        response = http.request(get)
-        json = JSON.parse(response.body) rescue nil if Net::HTTPOK === response
-        unless json
-          logger.error "Backtweets: #{response.code} #{response.message}"
-          raise "Last request didn't go as expected, trying again later"
-        end
-       
-        if tweets = json["tweets"]
-          last_tweet_id = source["last_tweet_id"]
-          if last_tweet_id
+    def update(source, request, callback)
+      session "backtweets.com" do |http|
+        http.get_json "/search.json?key=#{api_key}&q=#{Rack::Utils.escape source["url"]}" do |status, json|
+          if status == 200 && tweets = json["tweets"]
+            last_tweet_id = source["last_tweet_id"]
             new_ones = tweets.take_while { |tweet| tweet["tweet_id"] != last_tweet_id }.reverse
             new_ones.each do |tweet|
-              handle, id = tweet["tweet_from_user"], tweet["tweet_id"]
-              url = "http://twitter.com/#{handle}/#{id}"
+              screen_name, id = tweet["tweet_from_user"], tweet["tweet_id"]
+              url = "http://twitter.com/#{screen_name}/#{id}"
               html = <<-HTML
 <a href="#{url}">tweeted</a>:
 <blockquote>#{tweet["tweet_text"]}</blockquote>
               HTML
-              person = { :fullname=>handle, :identities=>%W{twitter.com:#{handle}}, :photo_url=>tweet["tweet_profile_image_url"] }
-              block.call :activity=>{ :uid=>id, :url=>url, :html=>html, :tags=>%w{twitter mention},
-                                      :timestamp=>Time.parse(tweet["tweet_created_at"]).utc, :person=>person }
+              person = { fullname: screen_name, identities: %W{twitter.com:#{screen_name}},
+                         photo_url: "http://img.tweetimag.es/i/#{screen_name}_n" }
+              callback.activity! uid: id, url: url, html: html, tags: %w{twitter mention},
+                                 timestamp: Time.parse(tweet["tweet_created_at"]).utc, person: person
+              source["last_tweet_id"] = tweet["tweet_id"]
             end
-          end
-          if recent = tweets.first
-            source["last_tweet_id"] = recent["tweet_id"]
+            callback.set! tweets: json["totalresults"].to_i
+          else
+            callback.error! "Last request didn't go as expected, trying again later"
           end
         end
-
-        block.call :set=>{ :tweets=>json["totalresults"].to_i }
       end
     end
 
     def meta(source)
-      [ { :text=>"Search yourself", :url=>"http://backtweets.com/search?q=#{URI.escape source["url"]}" } ]
+      [ { text: "Search yourself", url: "http://backtweets.com/search?q=#{URI.escape source["url"]}" } ]
     end
   end
 end

data/lib/dash-fu/marios/github.rb

@@ -6,7 +6,7 @@ module DashFu::Mario
     def setup(source, params)
       repo = params["repo"].strip
       source["source.name"] = "Github: #{repo}"
-      source["metric.columns"] = [{ :id=>"commits", :label=>"Commits" }, { :id=>"watchers", :label=>"Watchers" }, { :id=>"forks", :label=>"Forks" }]
+      source["metric.columns"] = [{ id: "commits", label: "Commits" }, { id: "watchers", label: "Watchers" }, { id: "forks", label: "Forks" }]
       source["metric.totals"] = true
       source["repo"] = repo
       branch = params["branch"].strip
@@ -20,80 +20,73 @@ module DashFu::Mario
       raise "Need user name and repository name, e.g. assaf/vanity" unless source["repo"][/^[\w-]+\/[\w-]+$/]
     end
 
-    def update(source, request, &block)
-      http = Net::HTTP.new("github.com", 443)
-      http.use_ssl = true
-      http.start do
-        case response = http_request(http, source, "/api/v2/json/repos/show/:repo")
-        when Net::HTTPOK
-          json = JSON.parse(response.body)["repository"] rescue nil
-        when Net::HTTPNotFound, Net::HTTPBadRequest
-          raise "Could not find the repository #{source["repo"]}"
-        when Net::HTTPUnauthorized
-          raise "You are not authorized to access this repository, or invalid username/password"
-        end
-        if json
-          source.update json.slice(*%w{description url homepage})
-          block.call :set=>{ :forks=>json["forks"], :watchers=>json["watchers"] }
-        else
-          logger.error "Github: #{response.code} #{response.message}"
-          error = true
-        end
-
-        case response = http_request(http, source, "/api/v2/json/commits/list/:repo/:branch")
-        when Net::HTTPOK
-          commits = JSON.parse(response.body)["commits"] rescue nil
-        when Net::HTTPNotFound, Net::HTTPBadRequest
-          raise "Could not find the branch #{source["branch"]}"
+    def update(source, request, callback)
+      session "github.com", 443 do |http|
+        auth = { username: "#{source["username"]}/token", password: source["api_token"] }
+        http.get_json "/api/v2/json/repos/show/#{source["repo"]}", auth do |status, json|
+          case status
+          when 200
+            if repository = json["repository"]
+              source.update repository.slice(*%w{description url homepage})
+              callback.set! forks: repository["forks"], watchers: repository["watchers"]
+            end
+          when 404, 400
+            callback.error! "Could not find the repository #{source["repo"]}"
+          when 401
+            callback.error! "You are not authorized to access this repository, or invalid username/password"
+          else
+            callback.error! "Last request didn't go as expected, trying again later"
+          end
         end
 
-        if commits
-          last_seen_id = source["last_commit"]["id"] if source["last_commit"]
-          if last_seen_id
-            new_ones = commits.take_while { |commit| commit["id"] != last_seen_id }
-            merged = new_ones.inject([]) do |all, commit|
-              last = all.last.last unless all.empty?
-              if last && last["committer"]["email"] == commit["committer"]["email"]
-                all.last << commit
-              else
-                all << [commit]
+        http.get_json "/api/v2/json/commits/list/#{source["repo"]}/#{source["branch"]}", auth do |status, json|
+          case status
+          when 200
+            if commits = json["commits"]
+              last_seen_id = source["last_commit"]["id"] if source["last_commit"]
+              new_ones = commits.take_while { |commit| commit["id"] != last_seen_id }
+              merged = new_ones.inject([]) do |all, commit|
+                last = all.last.last unless all.empty?
+                if last && last["committer"]["email"] == commit["committer"]["email"] &&
+                  Time.parse(last["committed_date"]) - Time.parse(commit["committed_date"]) < 1.hour
+                  all.last << commit
+                else
+                  all << [commit]
+                end
+                all
               end
-              all
-            end
 
-            merged.reverse.each do |commits|
-              first = commits.first
-              committer = first["committer"]
-              person = { :fullname=>committer["name"], :identities=>%W{github.com:#{committer["login"]}}, :email=>committer["email"] }
-              messages = commits.map { |commit| %{<blockquote><a href="#{commit["url"]}">#{commit["id"][0,7]}</a> #{h commit["message"].strip.split(/[\n\r]/).first[0,50]}</blockquote>} }
-              html = %{pushed to #{h source["branch"]} at <a href="http://github.com/#{source["repo"]}">#{h source["repo"]}</a>:\n#{messages.join("\n")}}
-              block.call :activity=>{ :uid=>first["id"], :html=>html, :url=>first["url"], :tags=>%w{push},
-                                      :timestamp=>Time.parse(first["committed_date"]).utc, :person=>person }
+              merged.reverse.each do |commits|
+                first = commits.first
+                committer = first["committer"]
+                person = { fullname: committer["name"], identities: %W{github.com:#{committer["login"]}}, email: committer["email"] }
+                messages = commits.map { |commit| %{<blockquote><a href="#{commit["url"]}">#{commit["id"][0,7]}</a> #{h commit["message"].strip.split(/[\n\r]/).first[0,50]}</blockquote>} }
+                html = %{pushed to #{h source["branch"]} at <a href="http://github.com/#{source["repo"]}">#{h source["repo"]}</a>:\n#{messages.join("\n")}}
+                callback.activity! uid: first["id"], html: html, url: first["url"], tags: %w{push},
+                                   timestamp: Time.parse(first["committed_date"]).utc, person: person
+              end
+              callback.inc! commits: new_ones.count
+              if last_commit = commits.first
+                source.update "last_commit"=>last_commit.slice("id", "message", "url").
+                  merge("timestamp"=>Time.parse(last_commit["committed_date"]).utc)
+              end
             end
-            block.call :inc=>{ :commits=>new_ones.count }
+          when 404, 400
+            callback.error! "Could not find the branch #{source["branch"]}"
           else
-            block.call :set=>{ :commits=>commits.count }
+            callback.error! "Github: #{response.code} #{response.message}"
           end
-          if last_commit = commits.first
-            source.update "last_commit"=>last_commit.slice("id", "message", "url").
-              merge("timestamp"=>Time.parse(last_commit["committed_date"]).utc)
-          end
-        else
-          logger.error "Github: #{response.code} #{response.message}"
-          error = true
         end
-
-        raise "Last request didn't go as expected, trying again later" if error
       end
     end
 
     def meta(source)
-      meta = [ { :title=>"Repository", :text=>source["repo"], :url=>source["url"] },
-               { :title=>"Branch", :text=>source["branch"] },
-               { :text=>source["description"] },
-               { :title=>"Home page", :url=>source["homepage"] } ]
+      meta = [ { title: "Repository", text: source["repo"], url: source["url"] },
+               { title: "Branch", text: source["branch"] },
+               { text: source["description"] },
+               { title: "Home page", url: source["homepage"] } ]
       if last_commit = source["last_commit"]
-        meta << { :title=>"Commit", :text=>last_commit["message"], :url=>last_commit["url"] }
+        meta << { title: "Commit", text: last_commit["message"], url: last_commit["url"] }
       end
       meta
     end

data/lib/dash-fu/marios/github_issues.rb

@@ -6,7 +6,7 @@ module DashFu::Mario
     def setup(source, params)
       repo = params["repo"].to_s.strip
       source["source.name"] = "Github Issues for #{repo}"
-      source["metric.columns"] = [{ :id=>"open", :label=>"Open issues" }, { :id=>"closed", :label=>"Closed issues" }]
+      source["metric.columns"] = [{ id: "open", label: "Open issues" }, { id: "closed", label: "Closed issues" }]
       source["metric.totals"] = true
       source["repo"] = repo
       source["username"] = params["username"]
@@ -18,64 +18,63 @@ module DashFu::Mario
       raise "Need user name and repository name, e.g. assaf/vanity" unless source["repo"][/^[\w-]+\/[\w-]+$/]
     end
 
-    def update(source, request, &block)
-      http = Net::HTTP.new("github.com", 443)
-      http.use_ssl = true
-      http.start do
-        update = {}
-        case response = http_request(http, source, "open")
-        when Net::HTTPOK
-          open = JSON.parse(response.body)["issues"] rescue nil
-        when Net::HTTPNotFound, Net::HTTPBadRequest
-          raise "Could not find the repository #{source["repo"]}"
-        when Net::HTTPUnauthorized
-          raise "You are not authorized to access this repository, or invalid username/password"
-        end
-        if open
-          update[:open] = open.count
-          if open_ids = source["open-ids"]
-            open.reject { |issue| open_ids.include?(issue["number"]) }.reverse.each do |issue|
-              sha = Digest::SHA1.hexdigest([source["repo"], "open", issue["number"], issue["updated_at"]].join(":"))
-              url = "http://github.com/#{source["repo"]}/issues#issue/#{issue["number"]}"
-              html = <<-HTML
+    def update(source, request, callback)
+      session "github.com", 443 do |http|
+        auth = { username: "#{source["username"]}/token", password: source["api_token"] }
+        http.get_json "/api/v2/json/issues/list/#{source["repo"]}/open" do |status, json|
+          case status
+          when 200
+            if open = json["issues"]
+              callback.set! open: open.count
+              open_ids = Set.new(source["open_ids"])
+              open.reject { |issue| open_ids.include?(issue["number"]) }.reverse.each do |issue|
+                sha = Digest::SHA1.hexdigest([source["repo"], "open", issue["number"], issue["updated_at"]].join(":"))
+                url = "http://github.com/#{source["repo"]}/issues#issue/#{issue["number"]}"
+                html = <<-HTML
 opened <a href="#{url}">issue #{issue["number"]}</a> on #{source["repo"]}:
 <blockquote>#{h issue["title"]}</blockquote>
-              HTML
-              block.call :activity=>{ :uid=>sha, :url=>url, :html=>html, :tags=>%w{issue opened},
-                                      :timestamp=>Time.parse(issue["created_at"]).utc }
+                HTML
+                callback.activity! uid: sha, url: url, html: html, tags: %w{issue opened},
+                                   timestamp: Time.parse(issue["created_at"]).utc
+                open_ids << issue["number"]
+              end
+              source["open_ids"] = open_ids.to_a
             end
+          when 404, 400
+            callback.error! "Could not find the repository #{source["repo"]}"
+          when 401
+            callback.error! "You are not authorized to access this repository, or invalid username/password"
+          else
+            callback.error! "Last request didn't go as expected, trying again later"
           end
-          source["open-ids"] = open.map { |issue| issue["number"] }
         end
 
-        case response = http_request(http, source, "closed")
-        when Net::HTTPOK
-          closed = JSON.parse(response.body)["issues"] rescue nil
-        end
-        if closed
-          update[:closed] = closed.count
-          if closed_ids = source["closed-ids"]
-            closed.reject { |issue| closed_ids.include?(issue["number"]) }.reverse.each do |issue|
-              sha = Digest::SHA1.hexdigest([source["repo"], "closed", issue["number"], issue["updated_at"]].join(":"))
-              url = "http://github.com/#{source["repo"]}/issues#issue/#{issue["number"]}"
-              html = <<-HTML
+        http.get_json "/api/v2/json/issues/list/#{source["repo"]}/closed" do |status, json|
+          case status
+          when 200
+            if closed = json["issues"]
+              callback.set! closed: closed.count
+              closed_ids = Set.new(source["closed_ids"])
+              closed.reject { |issue| closed_ids.include?(issue["number"]) }.reverse.each do |issue|
+                sha = Digest::SHA1.hexdigest([source["repo"], "closed", issue["number"], issue["updated_at"]].join(":"))
+                url = "http://github.com/#{source["repo"]}/issues#issue/#{issue["number"]}"
+                html = <<-HTML
 closed <a href="#{url}">issue #{issue["number"]}</a> on #{source["repo"]}:
 <blockquote>#{h issue["title"]}</blockquote>
-              HTML
-              block.call :activity=>{ :uid=>sha, :url=>url, :html=>html, :tags=>%w{issue closed},
-                                      :timestamp=>Time.parse(issue["closed_at"]).utc }
+                HTML
+                callback.activity! uid: sha, url: url, html: html, tags: %w{issue closed},
+                                   timestamp: Time.parse(issue["closed_at"]).utc
+                closed_ids << issue["number"]
+              end
+              source["closed_ids"] = closed_ids.to_a
             end
           end
-          source["closed-ids"] = closed.map { |issue| issue["number"] }
         end
-
-        raise "Last request didn't go as expected, trying again later" if update.empty?
-        block.call :set=>update
       end
     end
 
     def meta(source)
-      [ { :title=>"On Github", :url=>"http://github.com/#{source["repo"]}/issues" } ]
+      [ { title: "On Github", url: "http://github.com/#{source["repo"]}/issues" } ]
     end
 
   protected