ghtorrent 0.4 → 0.5

data/CHANGELOG

@@ -0,0 +1,24 @@
+= Version 0.5
+
+    * Generic methods for retrieving items that are bound to repositories
+	* Processing of pull requests with commits, comments and history
+	* Processing of project forks
+	* New tool (ght-load) to filter and load events to the queue
+	* New tool (ght-rm-dupl) to delete duplicate entries from collections (events & commits supported)
+	* Project wide requesting result caching for multi-page requests
+	* Better logging in various places
+	* Better defaults for MySQL (UTF8 + InnoDB tables)
+	* Commits are now seperated from projects. Project forks can share commits.
+	* Support for setting the IP address to use for retrieval on multi-homed
+	* Compatibility with Ruby 1.9 (now default) and JRuby
+	* Proper modularization, following the cake design pattern
+	* Never retrieve arrays of results from MongoDB
+
+= Version 0.4 
+
+  * Implement support for retrieving watches and project members
+	* Support for processing FollowEvents, WatchEvents, CommitCommentEvents, MemberEvents
+	* Projects are exclusively associated to users
+	* Remove dependence on Github API v2
+	* Remove license headers
+

data/Gemfile

@@ -0,0 +1,17 @@
+source 'https://rubygems.org'
+
+gem "amqp"
+gem "mongo"
+gem "trollop"
+gem "sequel"
+gem "daemons"
+gem "json"
+
+platforms :ruby do
+  gem "sqlite3"
+  gem "bson_ext"
+end
+
+platforms :jruby do
+  gem "jdbc-mysql"
+end

data/Gemfile.lock

@@ -0,0 +1,40 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    amq-client (0.9.4)
+      amq-protocol (>= 0.9.4)
+      eventmachine
+    amq-protocol (0.9.4)
+    amqp (0.9.7)
+      amq-client (~> 0.9.4)
+      amq-protocol (>= 0.9.4)
+      eventmachine
+    bson (1.6.4)
+    bson (1.6.4-java)
+    bson_ext (1.6.4)
+      bson (~> 1.6.4)
+    daemons (1.1.8)
+    eventmachine (0.12.10)
+    eventmachine (0.12.10-java)
+    jdbc-mysql (5.1.13)
+    json (1.7.3)
+    mongo (1.6.4)
+      bson (~> 1.6.4)
+    sequel (3.37.0)
+    sqlite3 (1.3.6)
+    trollop (1.16.2)
+
+PLATFORMS
+  java
+  ruby
+
+DEPENDENCIES
+  amqp
+  bson_ext
+  daemons
+  jdbc-mysql
+  json
+  mongo
+  sequel
+  sqlite3
+  trollop

data/README.md

@@ -10,18 +10,18 @@ GHTorrent relies on the following software to work:
 * MongoDB > 2.0
 * RabbitMQ >= 2.7
 * An SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). 
-GHTorrent is tested with SQLite and MySQL, so your mileage may vary if you are using other databases.
+GHTorrent is tested mainly with MySQL, so your mileage may vary if you are using other databases.
 
-GHTorrent is written in Ruby (tested with 1.8 and JRuby). To install 
+GHTorrent is written in Ruby (tested with 1.9 and JRuby). To install 
 it as a Gem do:
 
 <code>
 sudo gem install ghtorrent 
 </code>
 
-Depending on which SQL database you want to use, install the appropriate dependency gem.
-GHTorrent already installs the `sqlite3` gem (if it fails, install the development
-package for `sqlite3` for your system).
+Depending on which SQL database you want to use, install the appropriate
+dependency gem. GHTorrent already installs the `sqlite3` gem (if it fails,
+install the development package for `sqlite3` for your system).
 
 <code>
 sudo gem install mysql2 #or postgres
@@ -35,10 +35,10 @@ file to a file in your home directory. All provided scripts accept the `-c`
 option, which you can use to pass the location of the configuration file as
 a parameter.
 
-Edit the MongoDB and AMQP 
-configuration options accordingly. The scripts require accounts with permissions
-to create queues and exchanges in the AMQP queue, collections
-in MongoDB and tables in the selected SQL database, respectively.
+Edit the MongoDB and AMQP configuration options accordingly. The scripts
+require accounts with permissions to create queues and exchanges in the AMQP
+queue, collections in MongoDB and tables in the selected SQL database,
+respectively.
 
 To prepare MongoDB:
 
@@ -76,25 +76,26 @@ to retrieve data in parallel on the [Wiki](https://github.com/gousiosg/github-mi
 
 ### Running
 
-To retrieve data with GHTorrent 
+To retrieve data with GHTorrent: 
 
 * `ght-mirror-events.rb` periodically polls Github's event
-queue (`https://api.github.com/events`), stores all new events in the `events`
-collection in MongoDB and posts them to the `github` exchange in RabbitMQ.
+queue (`https://api.github.com/events`), stores all new events in the
+`events` collection in MongoDB and posts them to the `github` exchange in
+RabbitMQ.
 
 * `ght-data_retrieval.rb` creates queues that route posted events to processor
 functions, which in turn use the appropriate Github API call to retrieve the
 linked contents, extract metadata to store in the SQL database and store the
-retrieved data in the appropriate collection in Mongo, to avoid further
-API calls. Data in the SQL database contain pointers (the MongoDB key) 
-to the "raw" data in MongoDB.
-
-Both scripts can be run concurrently on more than one hosts, for resilience and
-performance reasons. To catch up with Github's event stream, it is enough to
-run `mirror_events.rb` on one host. To collect all data pointed by each event,
-one instance of `data_retrieval.rb` is not enough. Both scripts employ
-throttling mechanisms to keep API usage whithin the limits imposed by Github
-(currently 5000 reqs/hr).
+retrieved data in the appropriate collection in Mongo, to avoid further API
+calls. Data in the SQL database contain pointers (the MongoDB key) to the
+"raw" data in MongoDB.
+
+Both scripts can be run concurrently on more than one hosts, for resilience
+and performance reasons. To catch up with Github's event stream, it is
+usually enough to run `ght-mirror-events` on one host. To collect all data
+pointed by each event, one instance of `ght-data-retrieval` is not enough.
+Both scripts employ throttling mechanisms to keep API usage whithin the
+limits imposed by Github (currently 5000 reqs/hr).
 
 #### Data
 

data/bin/ght-data-retrieval

@@ -1,30 +1,33 @@
+#!/usr/bin/env ruby
+
 require 'rubygems'
 require 'amqp'
 require 'json'
-require 'ghtorrent'
 require 'pp'
 
+require 'ghtorrent/ghtorrent'
+require 'ghtorrent/settings'
+require 'ghtorrent/logging'
+require 'ghtorrent/command'
+
 class GHTDataRetrieval < GHTorrent::Command
 
   include GHTorrent::Settings
   include GHTorrent::Logging
 
-  attr_reader :settings, :name
-
   def parse(msg)
     JSON.parse(msg)
   end
 
-  def PushEvent(evt)
-    data = parse evt
+  def PushEvent(data)
     data['payload']['commits'].each do |c|
       url = c['url'].split(/\//)
+
       @gh.get_commit url[4], url[5], url[7]
     end
   end
 
-  def WatchEvent(evt)
-    data = parse evt
+  def WatchEvent(data)
     owner = data['repo']['name'].split(/\//)[0]
     repo = data['repo']['name'].split(/\//)[1]
     watcher = data['actor']['login']
@@ -33,8 +36,7 @@ class GHTDataRetrieval < GHTorrent::Command
     @gh.get_watcher owner, repo, watcher, created_at
   end
 
-  def FollowEvent(evt)
-    data = parse evt
+  def FollowEvent(data)
     follower = data['actor']['login']
     followed = data['payload']['target']['login']
     created_at = data['created_at']
@@ -42,8 +44,7 @@ class GHTDataRetrieval < GHTorrent::Command
     @gh.get_follower(follower, followed, created_at)
   end
 
-  def MemberEvent(evt)
-    data = parse evt
+  def MemberEvent(data)
     owner = data['actor']['login']
     repo = data['repo']['name'].split(/\//)[1]
     new_member = data['payload']['member']['login']
@@ -52,8 +53,7 @@ class GHTDataRetrieval < GHTorrent::Command
     @gh.get_project_member(owner, repo, new_member, created_at)
   end
 
-  def CommitCommentEvent(evt)
-    data = parse evt
+  def CommitCommentEvent(data)
     user = data['actor']['login']
     repo = data['repo']['name'].split(/\//)[1]
     id = data['payload']['comment']['id']
@@ -62,25 +62,63 @@ class GHTDataRetrieval < GHTorrent::Command
     @gh.get_commit_comment(user, repo, id, created_at)
   end
 
+  def PullRequestEvent(data)
+    owner = data['payload']['pull_request']['base']['repo']['owner']['login']
+    repo = data['payload']['pull_request']['base']['repo']['name']
+    pullreq_id = data['payload']['number']
+    action = data['payload']['action']
+    created_at = data['created_at']
+
+    @gh.get_pull_request(owner, repo, pullreq_id, action, created_at)
+  end
+
+  def ForkEvent(data)
+    owner = data['repo']['name'].split(/\//)[0]
+    repo = data['repo']['name'].split(/\//)[1]
+    fork_id = data['payload']['forkee']['id']
+    created_at = data['created_at']
+
+    @gh.get_fork(owner, repo, fork_id, created_at)
+  end
+
+  def PullRequestReviewCommentEvent(data)
+    owner = data['repo']['name'].split(/\//)[0]
+    repo = data['repo']['name'].split(/\//)[1]
+    comment_id = data['payload']['comment']['id']
+    pullreq_id = data['payload']['comment']['_links']['pull_request']['href'].split(/\//)[-1]
+    created_at = data['created_at']
+
+    @gh.get_pullreq_comment(owner, repo, pullreq_id, comment_id, created_at)
+  end
+
+  def IssueCommentEvent(data)
+    owner = data['repo']['name'].split(/\//)[0]
+    repo = data['repo']['name'].split(/\//)[1]
+    pullreq_id = data['payload']['forkee']['id']
+    created_at = data['created_at']
+
+    @gh.get_issue_comment(owner, repo, issue_id, comment_id, created_at)
+  end
+
   def handlers
-    %w(PushEvent WatchEvent FollowEvent MemberEvent CommitCommentEvent)
+    %w(PushEvent WatchEvent FollowEvent MemberEvent CommitCommentEvent PullRequestEvent ForkEvent PullRequestReviewCommentEvent)
+    #%w(PullRequestReviewCommentEvent)
   end
 
-  def prepare_options(options)
-      @name = "ght-data-retrieval"
+  def logger
+    @gh.logger
   end
 
   def go
-    @gh = GHTorrent::Mirror.new(options[:config])
-    @settings = @gh.settings
+    @gh = GHTorrent::Mirror.new(@settings)
 
     # Graceful exit
     Signal.trap('INT') {
-      info "Received SIGINT, exiting"
+      info "GHTDataRetrieval: Received SIGINT, exiting"
       AMQP.stop { EM.stop }
     }
     Signal.trap('TERM') {
-      info "Received SIGTERM, exiting"
+      info "GHTDataRetrieval: Received SIGTERM, exiting"
       AMQP.stop { EM.stop }
     }
 
@@ -89,7 +127,7 @@ class GHTDataRetrieval < GHTorrent::Command
                :username => config(:amqp_username),
                :password => config(:amqp_password)) do |connection|
 
-      channel = AMQP::Channel.new(connection, :prefetch => 5)
+      channel = AMQP::Channel.new(connection, :prefetch => config(:amqp_prefetch))
       exchange = channel.topic(config(:amqp_exchange), :durable => true,
                                :auto_delete => false)
 
@@ -97,21 +135,25 @@ class GHTDataRetrieval < GHTorrent::Command
         queue = channel.queue("#{h}s", {:durable => true})\
                        .bind(exchange, :routing_key => "evt.#{h}")
 
-        info "Binding handler #{h} to routing key evt.#{h}"
+        info "GHTDataRetrieval: Binding handler #{h} to routing key evt.#{h}"
 
         queue.subscribe(:ack => true) do |headers, msg|
           begin
-            send(h, msg)
+            data = parse(msg)
+            info "GHTDataRetrieval: Processing event: #{data['type']}-#{data['id']}"
+	          send(h, data)
             headers.ack
+            info "GHTDataRetrieval: Processed event: #{data['type']}-#{data['id']}"
           rescue Exception => e
             # Give a message a chance to be reprocessed
             if headers.redelivered?
+              data = parse(msg)
+              warn "GHTDataRetrieval: Could not process event: #{data['type']}-#{data['id']}"
               headers.reject(:requeue => false)
             else
               headers.reject(:requeue => true)
             end
 
-            #pp JSON.parse(msg)
             STDERR.puts e
             STDERR.puts e.backtrace.join("\n")
           end

data/bin/ght-load

@@ -1,35 +1,48 @@
+#!/usr/bin/env ruby
+
 require 'rubygems'
-require 'ghtorrent-old'
 require 'mongo'
 require 'amqp'
 require 'set'
 require 'eventmachine'
-require 'optparse'
-require 'ostruct'
 require 'pp'
 require "amqp/extensions/rabbitmq"
 
+require 'ghtorrent/settings'
+require 'ghtorrent/logging'
+require 'ghtorrent/persister'
+require 'ghtorrent/command'
+require 'ghtorrent/bson_orderedhash'
+
 class GHTLoad < GHTorrent::Command
 
+  include GHTorrent::Settings
+  include GHTorrent::Persister
+
   def col_info()
     {
         :commits => {
             :name => "commits",
             :payload => "commit.id",
             :unq => "commit.id",
-            :col => GH.commits_col,
+            :col => persister.get_underlying_connection.collection(:commits.to_s),
             :routekey => "commit.%s"
         },
         :events => {
             :name => "events",
             :payload => "",
             :unq => "type",
-            :col => GH.events_col,
+            :col => persister.get_underlying_connection.collection(:events.to_s),
             :routekey => "evt.%s"
         }
     }
   end
 
+  def persister
+    @persister ||= connect(:mongo, @settings)
+    @persister
+  end
+
   def prepare_options(options)
     options.banner <<-BANNER
 Loads object ids from a collection to a queue for further processing.
@@ -41,6 +54,8 @@ Loads object ids from a collection to a queue for further processing.
 
     options.opt :earliest, 'Seconds since epoch of earliest item to load',
       :short => 'e', :default => 0, :type => :int
+    options.opt :number, 'Number of items to load (-1 means all)',
+                :short => 'n', :type => :int, :default => -1
     options.opt :filter,
                 'Filter items by regexp on item attributes: item.attr=regexp',
                 :short => 'f', :type => String, :multi => true
@@ -63,10 +78,6 @@ Loads object ids from a collection to a queue for further processing.
   end
 
   def go
-    @gh = GHTorrent::Mirror.new(options[:config])
-    @settings = @gh.settings
-
-    GH.init(options[:config])
     # Message tags await publisher ack
     awaiting_ack = SortedSet.new
 
@@ -82,6 +93,7 @@ Loads object ids from a collection to a queue for further processing.
 
     puts "Loading from collection #{collection}"
     puts "Loading items after #{Time.at(options[:earliest])}" if options[:verbose]
+    puts "Loading #{options[:number]} items" if options[:verbose] && options[:number] != -1
 
     what = case
              when options[:filter].is_a?(Array)
@@ -98,13 +110,13 @@ Loads object ids from a collection to a queue for further processing.
 
     (puts "Mongo filter:"; pp what.merge(from)) if options[:verbose]
 
-    AMQP.start(:host => GH.settings['amqp']['host'],
-               :port => GH.settings['amqp']['port'],
-               :username => GH.settings['amqp']['username'],
-               :password => GH.settings['amqp']['password']) do |connection|
+    AMQP.start(:host => config(:amqp_host),
+               :port => config(:amqp_port),
+               :username => config(:amqp_username),
+               :password => config(:amqp_password)) do |connection|
 
       channel = AMQP::Channel.new(connection)
-      exchange = channel.topic(GH.settings['amqp']['exchange'],
+      exchange = channel.topic(config(:amqp_exchange),
                                :durable => true, :auto_delete => false)
 
       # What to do when the user hits Ctrl+c
@@ -115,18 +127,28 @@ Loads object ids from a collection to a queue for further processing.
       # Read next 1000 items and queue them
       read_and_publish = Proc.new {
 
+        to_read = if options.number == -1
+                    1000
+                  else
+                    if options.number - num_read - 1 <= 0
+                      -1
+                    else
+                      options.number - num_read - 1
+                    end
+                  end
+
         read = 0
         col_info[collection][:col].find(what.merge(from),
                                        :skip => num_read,
-                                       :limit => 1000).each do |e|
+                                       :limit => to_read).each do |e|
 
-          payload = GH.read_value(e, col_info[collection][:payload])
+          payload = read_value(e, col_info[collection][:payload])
           payload = if payload.class == BSON::OrderedHash
                       payload.delete "_id" # Inserted by MongoDB on event insert
                       payload.to_json
                     end
           read += 1
-          unq = GH.read_value(e, col_info[collection][:unq])
+          unq = read_value(e, col_info[collection][:unq])
           if unq.class != String or unq.nil? then
             throw Exception("Unique value can only be a String")
           end
@@ -136,12 +158,12 @@ Loads object ids from a collection to a queue for further processing.
           exchange.publish payload, :persistent => true, :routing_key => key
 
           num_read += 1
-          puts("Publish id = #{unq} (#{num_read} total)") if options.verbose
+          puts("Publish id = #{payload[unq]} (#{num_read} total)") if options.verbose
           awaiting_ack << num_read
         end
 
         # Nothing new in the DB and no msgs waiting ack
-        if read == 0 and awaiting_ack.size == 0
+        if (read == 0 and awaiting_ack.size == 0) or to_read == -1
           puts("Finished reading, exiting")
           show_stopper.call
         end