ghtorrent 0.8 → 0.8.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 964930d19ab5d35f6c70cafe72c8abf0e682ac87
+  data.tar.gz: 7bda9c3d934ab7c3d292587991dedc3f6e762cbc
+SHA512:
+  metadata.gz: 591e92ae026990ba6a2f0dddfc32b47df8983b829d1b2d675f6fe9fde977fc063cce322649e7c3b05398bc891568c6ab11117c253a4f19ff7edc6a9aa1045129
+  data.tar.gz: f4ba04030bdec8a390dc800e676fbb1dfd2c6f0efc26ce4696f8b2e14a0ed3c1a7975f98c1e670fbaa89c8e3b5e0c8e96d171e88cf9a70dea6b99a71a28d351b

data/CHANGELOG

@@ -1,3 +1,12 @@
+= Version 0.8.1
+	* New tool to retrieve specific entities and their dependencies
+	* New tool to retrieve repositories en masse
+	* Support for resuming when exception occurs while processing items in loops
+	* Support for finer grained transactions when processing large entities
+	* Commit comments are now indexed per owner/repo (was just by comment id)
+	* Remove the unused daemon mode
+    * Various exception fixes and more detailed logging
+
 = Version 0.8
 	* Retrieve and process issue labels
 	* Retrive and process actors for pull request events

data/Gemfile

@@ -3,5 +3,5 @@ source 'https://rubygems.org'
 gemspec
 
 platforms :jruby do
-  gem "jdbc-mysql"
+  gem 'jdbc-mysql'
 end

data/Gemfile.lock

@@ -1,34 +1,44 @@
 PATH
   remote: .
   specs:
-    ghtorrent (0.7.3)
-      amqp (~> 1.0.0)
-      bson_ext (~> 1.8.0)
-      daemons (~> 1.1.0)
-      mongo (~> 1.8.0)
-      sequel (~> 3.47)
+    ghtorrent (0.8.1)
+      amqp (~> 1.1.0)
+      bson_ext (~> 1.9.0)
+      mongo (~> 1.9.0)
+      sequel (~> 4.5.0)
       trollop (~> 2.0.0)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    amq-client (1.0.2)
-      amq-protocol (>= 1.2.0)
+    addressable (2.3.5)
+    amq-protocol (1.9.0)
+    amqp (1.1.7)
+      amq-protocol (>= 1.9.0)
       eventmachine
-    amq-protocol (1.5.0)
-    amqp (1.0.2)
-      amq-client (~> 1.0.2)
-      amq-protocol (>= 1.3.0)
-      eventmachine
-    bson (1.8.5)
-    bson_ext (1.8.5)
-      bson (~> 1.8.5)
-    daemons (1.1.9)
+    bson (1.9.2)
+    bson_ext (1.9.2)
+      bson (~> 1.9.2)
+    crack (0.4.1)
+      safe_yaml (~> 0.9.0)
+    diff-lcs (1.2.5)
     eventmachine (1.0.3)
-    mongo (1.8.5)
-      bson (~> 1.8.5)
-    sequel (3.47.0)
+    mongo (1.9.2)
+      bson (~> 1.9.2)
+    rspec (2.14.1)
+      rspec-core (~> 2.14.0)
+      rspec-expectations (~> 2.14.0)
+      rspec-mocks (~> 2.14.0)
+    rspec-core (2.14.7)
+    rspec-expectations (2.14.4)
+      diff-lcs (>= 1.1.3, < 2.0)
+    rspec-mocks (2.14.4)
+    safe_yaml (0.9.7)
+    sequel (4.5.0)
     trollop (2.0)
+    webmock (1.16.0)
+      addressable (>= 2.2.7)
+      crack (>= 0.3.2)
 
 PLATFORMS
   ruby
@@ -36,3 +46,5 @@ PLATFORMS
 DEPENDENCIES
   ghtorrent!
   jdbc-mysql
+  rspec (~> 2.14.0)
+  webmock (~> 1.16)

data/README.md

@@ -1,4 +1,4 @@
-##ghtorrent: Mirror and process data from the Github API 
+# ghtorrent: Mirror and process data from the Github API 
 
 A library and a collection of scripts used to retrieve data from the Github API
 and extract metadata in an SQL database, in a modular and scalable manner. The
@@ -12,8 +12,10 @@ GHTorrent can be used for a variety of purposes, such as:
 * Create a queriable metadata index for a specific repository
 * Query the Github API using intelligent caching to avoid duplicate queries
 
-GHTorrent is comprised from the following components (which can be used
-individually):
+
+## Components
+
+GHTorrents components (which can be used individually) are:
 
 * [APIClient](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/api_client.rb): Knows how to query the Github API (both single entities and
 pages) and respect the API request limit. Can be configured to override the
@@ -26,56 +28,65 @@ store must support arbitrary queries to the stored JSON objects.
 * [GHTorrent](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/ghtorrent.rb): Knows how to extract information from the data retrieved by
 the retriever in order to update an SQL database (see [schema](http://ghtorrent.org/relational.html)) with metadata.
 
+### Component Configuration 
+
 The Persister and GHTorrent components have configurable back ends:
 
-* Persister: Either uses MongoDB > 2.0 (`mongo` driver) or no persister (`noop` driver)
-* GHTorrent: GHTorrent is tested mainly with MySQL, but can theoretically be
+* **Persister:** Either uses MongoDB > 2.0 (`mongo` driver) or no persister (`noop` driver)
+* **GHTorrent:** GHTorrent is tested mainly with MySQL, but can theoretically be
 used with any SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). Your milaege may vary.
 
-
 The distributed mirroring scripts also require RabbitMQ >= 2.8 or other
 
-#### Installing
 
+## Installation
+
+
+### 1. Install GHTorrent
 GHTorrent is written in Ruby (tested with 1.9). To install it as a Gem do:
 
 <code>
 sudo gem install ghtorrent 
 </code>
 
+
+### 2. Install Your Preferred Database
+
 Depending on which SQL database you want to use, install the appropriate
 dependency gem.
 
 <code>
-sudo gem install mysql2 #or sqlite3-ruby #or postgres
+sudo gem install mysql2 # or <sqlite3-ruby|postgres>
 </code>
 
-#### Configuring
 
-Copy the contents of the 
-[config.yaml.tmpl](https://github.com/gousiosg/github-mirror/blob/master/config.yaml.tmpl)
-file to a file in your home directory. All provided scripts accept the `-c`
-option, which you can use to pass the location of the configuration file as
+## Configuration
+
+Copy [config.yaml.tmpl](https://github.com/gousiosg/github-mirror/blob/master/config.yaml.tmpl)
+to a file in your home directory. 
+
+All provided scripts accept the `-c` option, which accepts the location of the configuration file as
 a parameter.
 
 You can find more information of how you can setup a mirroring cluster of machines
 to retrieve data in parallel on the [Wiki](https://github.com/gousiosg/github-mirror/wiki/Setting-up-a-mirroring-cluster).
 
-### Running
+
+## Using GHTorrent
 
 To mirror the event stream and capture all data: 
 
 * `ght-mirror-events.rb` periodically polls Github's event
 queue (`https://api.github.com/events`), stores all new events in the
-configured pestister and posts them to the `github` exchange in
+configured pestister, and posts them to the `github` exchange in
 RabbitMQ.
 
 * `ght-data_retrieval.rb` creates queues that route posted events to processor
-functions, which in turn use the appropriate Github API call to retrieve the
-linked contents, extract metadata to store in the SQL database and store the
+functions. The functions use the appropriate Github API call to retrieve the
+linked contents, extract metadata (for database storage), and store the
 retrieved data in the appropriate collection in the persister, to avoid
-duplicate API
-calls. Data in the SQL database contain pointers (the `ext_ref_id` field) to the
+duplicate API calls. 
+Data in the SQL database contain pointers (the `ext_ref_id` field) to the
 "raw" data in the persister.
 
 To retrieve data for a repository or user:
@@ -89,27 +100,31 @@ To perform maintenance:
 the `ght-data-retrieval` script to reprocess them
 * `ght-get-more-commits` retrieves all commits for a specific repository
 
-#### Data
+
+### Data Torrents
 
 You can find torrents for retrieving data on the 
-[Available Torrents](https://ghtorrent.org/downloads.html) page. You can find two sets of data:
+[Available Torrents](https://ghtorrent.org/downloads.html) page. 
 
-* Raw events: Github's [event stream](https://api.github.com/events). These
+There are two sets of data:
+
+* **Raw events:** Github's [event stream](https://api.github.com/events). These
 are the roots for mirroring operations. The `ght-data-retrieval` crawler starts
 from an event and goes deep into the rabbit hole.
-* SQL dumps+Linked data: Data dumps from the SQL database and the corresponding
+* **SQL dumps + Linked data:** Data dumps from the SQL database and the corresponding
 MongoDB entities.
 
-#### Reporting bugs
 
-Please use the [Issue
-Tracker](https://github.com/gousiosg/github-mirror/issues) for reporting bugs
-and feature requests.
+## Bugs & Feature Requests
+
+Please tell us about features you'd like or bugs you've discovered on our 
+[Issue Tracker](https://github.com/gousiosg/github-mirror/issues).
 
-Patches, bug fixes etc are welcome. Please fork the repository and create
+Patches, bug fixes, etc are welcome. Please fork the repository and create
 a pull request when done fixing/implementing the new feature.
 
-#### Citation information
+
+## Citing GHTorrent in your Research
 
 If you find GHTorrent and the accompanying datasets useful in your research,
 please consider citing the following paper:
@@ -118,16 +133,17 @@ please consider citing the following paper:
 
 See also the following presentation:
 
-<iframe src="http://www.slideshare.net/slideshow/embed_code/13184524?rel=0" width="342" height="291" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC;border-width:1px 1px 0;margin-bottom:5px" allowfullscreen/> 
+<iframe src="http://www.slideshare.net/slideshow/embed_code/13184524?rel=0" width="342" height="291" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC;border-width:1px 1px 0;margin-bottom:5px" allowfullscreen /> 
 <div style="margin-bottom:5px"> <strong> <a href="http://www.slideshare.net/gousiosg/ghtorrent-githubs-data-from-a-firehose-13184524" title="GHTorrent: Github&#39;s Data from a Firehose" target="_blank">GHTorrent: Github&#39;s Data from a Firehose</a> </strong> </div>
 
-#### Authors
 
-[Georgios Gousios](http://istlab.dmst.aueb.gr/~george) <gousiosg@gmail.com>
+## Authors
+
+* [Georgios Gousios](http://istlab.dmst.aueb.gr/~george) <gousiosg@gmail.com>
+* [Diomidis Spinellis](http://www.dmst.aueb.gr/dds) <dds@aueb.gr>
 
-[Diomidis Spinellis](http://www.dmst.aueb.gr/dds) <dds@aueb.gr>
 
-#### License
+## License
 
 [2-clause BSD](http://www.opensource.org/licenses/bsd-license.php)
 

data/Rakefile

@@ -2,11 +2,11 @@ require 'rake'
 require 'rake/testtask'
 require 'rake/rdoctask'
 
-task :default => [:test, :rdoc]
+task :default => [:spec, :rdoc]
 
 desc "Run basic tests"
-Rake::TestTask.new(:test) do |t|
-  t.pattern = 'test/*_test.rb'
+Rake::TestTask.new(:spec) do |t|
+  t.pattern = 'spec/*_test.rb'
   t.verbose = true
   t.warning = true
 end

data/bin/ght-retrieve-dependents

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'ghtorrent'
+
+GHTRetrieveDependents.run(ARGV)

data/bin/ght-retrieve-repos

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'ghtorrent'
+
+GHTRetrieveRepos.run(ARGV)

data/lib/ghtorrent.rb

@@ -48,6 +48,7 @@ require 'ghtorrent/retriever'
 
 # SQL database fillup methods
 require 'ghtorrent/ghtorrent'
+require 'ghtorrent/transacted_ghtorrent'
 
 # Commands
 require 'ghtorrent/commands/ght_data_retrieval'
@@ -57,5 +58,7 @@ require 'ghtorrent/commands/ght_rm_dupl'
 require 'ghtorrent/commands/ght_load'
 require 'ghtorrent/commands/ght_retrieve_repo'
 require 'ghtorrent/commands/ght_retrieve_user'
+require 'ghtorrent/commands/ght_retrieve_dependents'
+require 'ghtorrent/commands/ght_retrieve_repos'
 
 # vim: set sta sts=2 shiftwidth=2 sw=2 et ai :

data/lib/ghtorrent/adapters/mongo_persister.rb

@@ -24,7 +24,7 @@ module GHTorrent
         :events                => %w(id),
         :users                 => %w(login),
         :commits               => %w(sha),
-        :commit_comments       => %w(repo user commit_id),
+        :commit_comments       => %w(commit_id id),
         :repos                 => %w(name owner.login),
         :repo_labels           => %w(repo owner),
         :repo_collaborators    => %w(repo owner login),
@@ -184,9 +184,9 @@ module GHTorrent
         idx_fields = v.reduce({}){|acc, x| acc.merge({x => 1})}
         if exists.nil?
           col.create_index(idx_fields, :background => true)
-          STDERR.puts "Creating index on #{collection}(#{v})"
+          STDERR.puts "Creating index on #{col}(#{v})"
         else
-          STDERR.puts "Index on #{collection}(#{v}) exists"
+          STDERR.puts "Index on #{col}(#{v}) exists"
         end
 
       end

data/lib/ghtorrent/api_client.rb

@@ -169,7 +169,7 @@ module GHTorrent
             end
 
         total = Time.now.to_ms - start_time.to_ms
-        debug "APIClient: Request: #{url} #{if from_cache then "from cache," else "(#{@remaining} remaining)," end} Total: #{total} ms"
+        debug "APIClient[#{@attach_ip}]: Request: #{url} #{if from_cache then "from cache," else "(#{@remaining} remaining)," end} Total: #{total} ms"
 
         contents
       rescue OpenURI::HTTPError => e
@@ -190,9 +190,9 @@ module GHTorrent
         end
       ensure
         if not from_cache and config(:respect_api_ratelimit) and @remaining < 10
-          sleep = (@reset - Time.now.to_i) / 60
-          debug "APIClient: Request limit reached, sleeping for #{sleep} min"
-          sleep(@reset - Time.now.to_i)
+          to_sleep = @reset - Time.now.to_i + 2
+          debug "APIClient: Request limit reached, sleeping for #{to_sleep} secs"
+          sleep(to_sleep)
         end
       end
     end
@@ -202,6 +202,8 @@ module GHTorrent
       @username   ||= config(:github_username)
       @passwd     ||= config(:github_passwd)
       @user_agent ||= config(:user_agent)
+      @remaining  ||= 10
+      @reset      ||= Time.now.to_i + 3600
 
       open_func ||= if @username.nil?
         lambda {|url| open(url, 'User-Agent' => @user_agent)}

data/lib/ghtorrent/command.rb

@@ -54,29 +54,6 @@ module GHTorrent
                                                      command.options[:password])
         end
 
-        if command.options[:daemon]
-          if Process.uid == 0
-            # Daemonize as a proper system daemon
-            Daemons.daemonize(:app_name   => File.basename($0),
-                              :dir_mode   => :system,
-                              :log_dir    => "/var/log",
-                              :backtrace  => true,
-                              :log_output => true)
-            STDERR.puts "Became a daemon"
-            # Change effective user id for the process
-            unless command.options[:user].nil?
-              Process.euid = Etc.getpwnam(command.options[:user]).uid
-            end
-          else
-            # Daemonize, but output in current directory
-            Daemons.daemonize(:app_name   => File.basename($0),
-                              :dir_mode   => :normal,
-                              :dir        => Dir.getwd,
-                              :backtrace  => true,
-                              :log_output => true)
-          end
-        end
-
         begin
           command.go
         rescue => e
@@ -107,10 +84,7 @@ Standard options:
         opt :verbose, 'verbose mode', :short => 'v'
         opt :addr, 'ip address to use for performing requests', :short => 'a',
             :type => String
-        opt :daemon, 'run as daemon', :short => 'd'
-        opt :user, 'run as the specified user (only when started as root)',
-            :short => 'u', :type => String
-        opt :username, 'Username at Github', :type => String
+        opt :username, 'Username at Github', :short => 's', :type => String
         opt :password, 'Password at Github', :type => String
       end
     end
@@ -163,7 +137,7 @@ Standard options:
 
     def override_config(config_file, setting, new_value)
       puts "Overriding configuration #{setting}=#{config(setting)} with cmd line #{new_value}"
-      merge_config_values({setting => new_value})
+      merge_config_values(config_file, {setting => new_value})
     end
 
     private

data/lib/ghtorrent/commands/ght_data_retrieval.rb

@@ -58,11 +58,12 @@ class GHTDataRetrieval < GHTorrent::Command
   end
 
   def CommitCommentEvent(data)
-    user = data['actor']['login']
+    user = data['repo']['name'].split(/\//)[0]
     repo = data['repo']['name'].split(/\//)[1]
     id = data['payload']['comment']['id']
+    sha = data['payload']['comment']['commit_id']
 
-    ghtorrent.get_commit_comment(user, repo, id)
+    ghtorrent.get_commit_comment(user, repo, sha, id)
   end
 
   def PullRequestEvent(data)

data/lib/ghtorrent/commands/ght_load.rb

@@ -60,7 +60,7 @@ Loads object ids from a collection to a queue for further processing.
 
   def go
     # Num events read
-    num_read = 0
+    total_read = 0
 
     puts "Loading items after #{Time.at(options[:earliest])}" if options[:verbose]
     puts "Loading items before #{Time.at(options[:latest])}" if options[:verbose]
@@ -107,9 +107,10 @@ Loads object ids from a collection to a queue for further processing.
 
       # Read next options[:batch] items and queue them
       read_and_publish = Proc.new {
-
+        num_read = 0
         persister.get_underlying_connection[:events].find(what.merge(from),
-                                        :skip => num_read,
+                                        :snapshot => true,
+                                        :skip => total_read,
                                         :limit => options[:batch]).each do |e|
           unq = read_value(e, 'type')
           if unq.class != String or unq.nil? then
@@ -120,10 +121,11 @@ Loads object ids from a collection to a queue for further processing.
                            :routing_key => "evt.#{e['type']}"
 
           num_read += 1
-          puts "Publish id = #{e['id']} (#{num_read} total)" if options.verbose
+          total_read += 1
+          puts "Publish id = #{e['id']} (#{num_read} read, #{total_read} total)" if options.verbose
         end
 
-        if num_read >= options[:number]
+        if total_read >= options[:number] or num_read == 0
           puts 'Finished reading, exiting'
           show_stopper.call
         else