ghtorrent 0.2

data/README.md

@@ -0,0 +1,132 @@
+github-mirror: Mirror and process the Github event steam
+=========================================================
+
+A collection of scripts used to mirror the Github event stream, for 
+research purposes. The scripts are distributed as a Gem (`ghtorrent`),
+but they can also be run by checking out this repository.
+
+GHTorrent relies on the following software to work:
+
+* MongoDB > 2.0
+* RabbitMQ >= 2.7
+* An SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). GHTorrent is tested with SQLite and MySQL,
+so your mileage may vary if you are using other databases.
+
+GHTorrent is written in Ruby (tested with 1.8 and JRuby). To install 
+it as a Gem do:
+
+<code>
+sudo gem install ghtorrent 
+</code>
+
+#### Configuring
+
+Copy the contents of the 
+[config.yaml.tmpl](https://github.com/gousiosg/github-mirror/blob/master/config.yaml.tmpl)
+file to a file in your home directory. All provided scripts accept the `-c`
+option, which you can use to pass the location of the configuration file as
+a parameter.
+
+Edit the MongoDB and AMQP 
+configuration options accordingly. The scripts require accounts with permissions
+to create queues and exchanges in the AMQP queue, collections
+in MongoDB and tables in the selected SQL database, respectively.
+
+To prepare MongoDB:
+
+<pre>
+$ mongo admin
+> db.addUser('github', 'github')
+> use github
+> db.addUser('github', 'github')
+</pre>
+
+To prepare RabbitMQ:
+
+<pre>
+$ rabbitmqctl add_user github
+$ rabbitmqctl set_permissions -p / github ".*" ".*" ".*"
+
+# The following will enable the RabbitMQ web admin for the github user
+# Not necessary to have, but good to debug and diagnose problems
+$ rabbitmq-plugins enable rabbitmq_management
+$ rabbitmqctl set_user_tags github administrator
+</pre>
+
+To prepare MySQL:
+
+<pre>
+$ mysql -u root -p
+mysql> create user 'github'@'localhost' identified by 'github';
+mysql> create database github;
+mysql> GRANT ALL PRIVILEGES ON github.* to github@'localhost';
+mysql> flush privileges;
+</pre>
+
+You can find more information of how you can setup a cluster of machines
+to retrieve data in parallel on the [Wiki](https://github.com/gousiosg/github-mirror/wiki/Setting-up-a-mirroring-cluster).
+
+### Running
+
+To retrieve data with GHTorrent 
+
+* `ght-mirror-events.rb` periodically polls Github's event
+queue (`https://api.github.com/events`), stores all new events in the `events`
+collection in MongoDB and posts them to the `github` exchange in RabbitMQ.
+
+* `ght-data_retrieval.rb` creates queues that route posted events to processor
+functions, which in turn use the appropriate Github API call to retrieve the
+linked contents, extract metadata to store in the SQL database and store the
+retrieved data in the appropriate collection in Mongo, to avoid further
+API calls. Data in the SQL database contain pointers (the MongoDB key) 
+to the "raw" data in MongoDB.
+
+Both scripts can be run concurrently on more than one hosts, for resilience and
+performance reasons. To catch up with Github's event stream, it is enough to
+run `mirror_events.rb` on one host. To collect all data pointed by each event,
+one instance of `data_retrieval.rb` is not enough. Both scripts employ
+throttling mechanisms to keep API usage whithin the limits imposed by Github
+(currently 5000 reqs/hr).
+
+#### Data
+
+You can find torrents for retrieving data on the 
+[Available Torrents](https://github.com/gousiosg/github-mirror/wiki/Available-Torrents) page. You need two sets of data:
+
+* Raw events: Github's [event stream](https://api.github.com/events). These
+are the roots for mirroring operations. The `ght-data-retrieval` crawler starts
+from an event and goes deep into the rabbit hole.
+* SQL dumps+Linked data: Data dumps from the SQL database and the corresponding
+MongoDB entities.
+
+
+*At the moment, GHTorrent is in the process of redesigning its data storage
+schema. Consequently, it does not distribute SQL dumps or linked data raw data.
+The distribution service will come back shortly.*
+
+#### Reporting bugs
+
+Please use the [Issue
+Tracker](https://github.com/gousiosg/github-mirror/issues) for reporting bugs
+and feature requests.
+
+Patches, bug fixes etc are welcome. Please fork the repository and create
+a pull request when done fixing/implementing the new feature.
+
+#### Citation information
+
+If you find GHTorrent and the accompanying datasets useful in your research,
+please consider citing the following paper:
+
+> Georgios Gousios and Diomidis Spinellis, "GHTorrent: GitHub’s data from a firehose," in _MSR '12: Proceedings of the 9th Working Conference on Mining Software Repositories_, June 2-–3, 2012. Zurich, Switzerland.
+
+#### Authors
+
+Georgios Gousios <gousiosg@gmail.com>
+
+Diomidis Spinellis
+
+#### License
+
+[2-clause BSD](http://www.opensource.org/licenses/bsd-license.php)
+

data/Rakefile

@@ -0,0 +1,20 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+task :default => [:test, :rdoc]
+
+desc "Run basic tests"
+Rake::TestTask.new(:test) do |t|
+  t.pattern = 'test/*_test.rb'
+  t.verbose = true
+  t.warning = true
+end
+
+desc "Run Rdoc"
+Rake::RDocTask.new(:rdoc) do |rd|
+#  rd.main = "README.doc"
+  rd.rdoc_files.include("lib/**/*.rb")
+  rd.options << "-d"
+  rd.options << "-x migrations"
+end

data/bin/ght-data-retrieval

@@ -0,0 +1,119 @@
+#!/usr/bin/env ruby
+
+# Copyright 2012 Georgios Gousios <gousiosg@gmail.com>
+#
+# Redistribution and use in source and binary forms, with or
+# without modification, are permitted provided that the following
+# conditions are met:
+#
+#   1. Redistributions of source code must retain the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer.
+#
+#   2. Redistributions in binary form must reproduce the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer in the documentation and/or other materials
+#      provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+require 'rubygems'
+require 'amqp'
+require 'json'
+require 'ghtorrent'
+require 'pp'
+
+class GHTDataRetrieval < GHTorrent::Command
+
+  include GHTorrent::Settings
+
+  attr_reader :settings
+
+  def parse(msg)
+    JSON.parse(msg)
+  end
+
+  def PushEvent(evt)
+    data = parse evt
+    data['payload']['commits'].each do |c|
+      url = c['url'].split(/\//)
+      @gh.get_commit url[4], url[5], url[7]
+    end
+  end
+
+  def WatchEvent(evt)
+    data = parse evt
+    user = data['actor']['login']
+    #@gh.get_watched user, evt
+  end
+
+  def FollowEvent(evt)
+    data = parse evt
+    user = data['actor']['login']
+    #@gh.get_followed user
+
+    followed = data['payload']['target']['login']
+    #@gh.get_followers followed
+  end
+
+  def handlers
+    %w(PushEvent WatchEvent FollowEvent)
+  end
+
+  def go
+    @gh = GHTorrent::Mirror.new(options[:config])
+    @settings = @gh.settings
+
+    # Graceful exit
+    Signal.trap('INT') { AMQP.stop { EM.stop } }
+    Signal.trap('TERM') { AMQP.stop { EM.stop } }
+
+    AMQP.start(:host => config(:amqp_host),
+               :port => config(:amqp_port),
+               :username => config(:amqp_username),
+               :password => config(:amqp_password)) do |connection|
+
+      channel = AMQP::Channel.new(connection, :prefetch => 5)
+      exchange = channel.topic(config(:amqp_exchange), :durable => true,
+                               :auto_delete => false)
+
+      handlers.each { |h|
+        queue = channel.queue("#{h}s", {:durable => true})\
+                       .bind(exchange, :routing_key => "evt.#{h}")
+
+        puts "Binding handler #{h} to routing key evt.#{h}"
+
+        queue.subscribe(:ack => true) do |headers, msg|
+          begin
+            send(h, msg)
+            headers.ack
+          rescue Exception => e
+            # Give a message a chance to be reprocessed
+            if headers.redelivered?
+              headers.reject(:requeue => false)
+            else
+              headers.reject(:requeue => true)
+            end
+
+            #pp JSON.parse(msg)
+            STDERR.puts e
+            STDERR.puts e.backtrace.join("\n")
+          end
+        end
+      }
+    end
+  end
+end
+
+GHTDataRetrieval.run

data/bin/ght-load

@@ -0,0 +1,242 @@
+#!/usr/bin/env ruby
+#
+# Loads items from Mongo to the queue for further processing
+#
+#
+# Copyright 2012 Georgios Gousios <gousiosg@gmail.com>
+#
+# Redistribution and use in source and binary forms, with or
+# without modification, are permitted provided that the following
+# conditions are met:
+#
+#   1. Redistributions of source code must retain the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer.
+#
+#   2. Redistributions in binary form must reproduce the above
+#      copyright notice, this list of conditions and the following
+#      disclaimer in the documentation and/or other materials
+#      provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+require 'rubygems'
+require 'ghtorrent-old'
+require 'mongo'
+require 'amqp'
+require 'set'
+require 'eventmachine'
+require 'optparse'
+require 'ostruct'
+require 'pp'
+require "amqp/extensions/rabbitmq"
+
+class GHTLoad < GHTorrent::Command
+
+  def col_info()
+    {
+        :commits => {
+            :name => "commits",
+            :payload => "commit.id",
+            :unq => "commit.id",
+            :col => GH.commits_col,
+            :routekey => "commit.%s"
+        },
+        :events => {
+            :name => "events",
+            :payload => "",
+            :unq => "type",
+            :col => GH.events_col,
+            :routekey => "evt.%s"
+        }
+    }
+  end
+
+  def prepare_options(options)
+    options.banner <<-BANNER
+Loads object ids from a collection to a queue for further processing.
+
+#{command_name} [options] collection
+
+#{command_name} options:
+    BANNER
+
+    options.opt :earliest, 'Seconds since epoch of earliest item to load',
+      :short => 'e', :default => 0, :type => :int
+    options.opt :filter,
+                'Filter items by regexp on item attributes: item.attr=regexp',
+                :short => 'f', :type => String, :multi => true
+  end
+
+  def validate
+    super
+    Trollop::die "no collection specified" unless args[0] && !args[0].empty?
+    filter = options[:filter]
+    case
+      when filter.is_a?(Array)
+        options[:filter].each { |x|
+          Trollop::die "not a valid filter #{x}" unless is_filter_valid?(x)
+        }
+      when filter == []
+        # Noop
+      else
+        Trollop::die "A filter can only be a string"
+    end
+  end
+
+  def go
+    @gh = GHTorrent::Mirror.new(options[:config])
+    @settings = @gh.settings
+
+    GH.init(options[:config])
+    # Message tags await publisher ack
+    awaiting_ack = SortedSet.new
+
+    # Num events read
+    num_read = 0
+
+    collection = case args[0]
+                   when "events"
+                     :events
+                   when "commits"
+                     :commits
+                 end
+
+    puts "Loading form collection #{collection}"
+    puts "Loading items after #{Time.at(options[:earliest])}" if options[:verbose]
+
+    what = case
+             when options[:filter].is_a?(Array)
+               options[:filter].reduce({}) { |acc,x|
+                 (k,r) = x.split(/=/)
+                 acc[k] = Regexp.new(r)
+                 acc
+               }
+             when filter == []
+               {}
+           end
+
+    from = {'_id' => {'$gte' => BSON::ObjectId.from_time(Time.at(options[:earliest]))}}
+
+    (puts "Mongo filter:"; pp what.merge(from)) if options[:verbose]
+
+    AMQP.start(:host => GH.settings['amqp']['host'],
+               :port => GH.settings['amqp']['port'],
+               :username => GH.settings['amqp']['username'],
+               :password => GH.settings['amqp']['password']) do |connection|
+
+      channel = AMQP::Channel.new(connection)
+      exchange = channel.topic(GH.settings['amqp']['exchange'],
+                               :durable => true, :auto_delete => false)
+
+      # What to do when the user hits Ctrl+c
+      show_stopper = Proc.new {
+        connection.close { EventMachine.stop }
+      }
+
+      # Read next 1000 items and queue them
+      read_and_publish = Proc.new {
+
+        read = 0
+        col_info[collection][:col].find(what.merge(from),
+                                       :skip => num_read,
+                                       :limit => 1000).each do |e|
+
+          payload = GH.read_value(e, col_info[collection][:payload])
+          payload = if payload.class == BSON::OrderedHash
+                      payload.delete "_id" # Inserted by MongoDB on event insert
+                      payload.to_json
+                    end
+          read += 1
+          unq = GH.read_value(e, col_info[collection][:unq])
+          if unq.class != String or unq.nil? then
+            throw Exception("Unique value can only be a String")
+          end
+
+          key = col_info[collection][:routekey] % unq
+
+          exchange.publish payload, :persistent => true, :routing_key => key
+
+          num_read += 1
+          puts("Publish id = #{unq} (#{num_read} total)") if options.verbose
+          awaiting_ack << num_read
+        end
+
+        # Nothing new in the DB and no msgs waiting ack
+        if read == 0 and awaiting_ack.size == 0
+          puts("Finished reading, exiting")
+          show_stopper.call
+        end
+      }
+
+      # Remove acknowledged or failed msg tags from the queue
+      # Trigger more messages to be read when ack msg queue size drops to zero
+      publisher_event = Proc.new { |ack|
+        if ack.multiple then
+          awaiting_ack.delete_if { |x| x <= ack.delivery_tag }
+        else
+          awaiting_ack.delete ack.delivery_tag
+        end
+
+        if awaiting_ack.size == 0
+          puts("ACKS.size= #{awaiting_ack.size}") if options.verbose
+          EventMachine.next_tick do
+            read_and_publish.call
+          end
+        end
+      }
+
+      # Await publisher confirms
+      channel.confirm_select
+
+      # Callback when confirms have arrived
+      channel.on_ack do |ack|
+        puts "ACK: tag=#{ack.delivery_tag}, mul=#{ack.multiple}" if options.verbose
+        publisher_event.call(ack)
+      end
+
+      # Callback when confirms failed.
+      channel.on_nack do |nack|
+        puts "NACK: tag=#{nack.delivery_tag}, mul=#{nack.multiple}" if options.verbose
+        publisher_event.call(nack)
+      end
+
+      # Signal handlers
+      Signal.trap('INT', show_stopper)
+      Signal.trap('TERM', show_stopper)
+
+      # Trigger start processing
+      EventMachine.add_timer(0.1) do
+        read_and_publish.call
+      end
+    end
+  end
+
+  private
+
+  def is_filter_valid?(filter)
+    (k, r) = filter.split(/=/)
+    return false if r.nil?
+    begin
+      Regexp.new(r)
+      true
+    rescue
+      false
+    end
+  end
+end
+
+GHTLoad.run
+
+#vim: set filetype=ruby expandtab tabstop=2 shiftwidth=2 autoindent smartindent: