ghtorrent 0.6 → 0.7.1

data/CHANGELOG

@@ -1,3 +1,15 @@
+= Version 0.7
+	* Full support for issues (comments, labels etc) and pull requests
+	* Cleaned up retrieval of pull request commits
+	* Cleaned up association of commits with repositories.
+	* Removed the forks table. Forks are now tracked by the forked_from field in projects
+	* Use Github's HTTP headers for request throttling
+	* Support for setting user agent header as per Github API requirements
+	* Support for marking projects as deleted (run fixes/update_deleted.rb)
+	* New tool (ght-retrieve-user) to retrieve all data for a single user 
+	* Support for running without a config.yaml file
+	* Technical report in doc/
+
 = Version 0.6
 
 	* Support retrieval of issues, issue events and issue history

data/Gemfile

@@ -1,16 +1,6 @@
 source 'https://rubygems.org'
 
-gem "amqp"
-gem "mongo"
-gem "trollop"
-gem "sequel"
-gem "daemons"
-gem "json"
-
-platforms :ruby do
-  gem "sqlite3"
-  gem "bson_ext"
-end
+gemspec
 
 platforms :jruby do
   gem "jdbc-mysql"

data/Gemfile.lock

@@ -1,40 +1,38 @@
+PATH
+  remote: .
+  specs:
+    ghtorrent (0.6)
+      amqp (~> 1.0.0)
+      bson_ext (~> 1.8.0)
+      daemons (~> 1.1.0)
+      mongo (~> 1.8.0)
+      sequel (~> 3.47)
+      trollop (~> 2.0.0)
+
 GEM
   remote: https://rubygems.org/
   specs:
-    amq-client (0.9.4)
-      amq-protocol (>= 0.9.4)
+    amq-client (1.0.2)
+      amq-protocol (>= 1.2.0)
       eventmachine
-    amq-protocol (0.9.4)
-    amqp (0.9.7)
-      amq-client (~> 0.9.4)
-      amq-protocol (>= 0.9.4)
+    amq-protocol (1.4.0)
+    amqp (1.0.2)
+      amq-client (~> 1.0.2)
+      amq-protocol (>= 1.3.0)
       eventmachine
-    bson (1.6.4)
-    bson (1.6.4-java)
-    bson_ext (1.6.4)
-      bson (~> 1.6.4)
-    daemons (1.1.8)
-    eventmachine (0.12.10)
-    eventmachine (0.12.10-java)
-    jdbc-mysql (5.1.13)
-    json (1.7.3)
-    mongo (1.6.4)
-      bson (~> 1.6.4)
-    sequel (3.37.0)
-    sqlite3 (1.3.6)
-    trollop (1.16.2)
+    bson (1.8.5)
+    bson_ext (1.8.5)
+      bson (~> 1.8.5)
+    daemons (1.1.9)
+    eventmachine (1.0.3)
+    mongo (1.8.5)
+      bson (~> 1.8.5)
+    sequel (3.47.0)
+    trollop (2.0)
 
 PLATFORMS
-  java
   ruby
 
 DEPENDENCIES
-  amqp
-  bson_ext
-  daemons
+  ghtorrent!
   jdbc-mysql
-  json
-  mongo
-  sequel
-  sqlite3
-  trollop

data/README.md

@@ -9,22 +9,21 @@ GHTorrent relies on the following software to work:
 
 * MongoDB > 2.0
 * RabbitMQ >= 2.7
-* An SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). 
-GHTorrent is tested mainly with MySQL, so your mileage may vary if you are using other databases.
+* MySQL >= 5.5. GHTorrent is tested mainly with MySQL, but can theoretically be
+used with any SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). Your milaege may vary.
 
-GHTorrent is written in Ruby (tested with 1.9 and JRuby). To install 
-it as a Gem do:
+GHTorrent is written in Ruby (tested with 1.9). To install it as a Gem do:
 
 <code>
 sudo gem install ghtorrent 
 </code>
 
 Depending on which SQL database you want to use, install the appropriate
-dependency gem. GHTorrent already installs the `sqlite3` gem (if it fails,
-install the development package for `sqlite3` for your system).
+dependency gem. GHTorrent already installs the `mysql2` gem (if it fails,
+install the development package for `libmysql-dev` for your system).
 
 <code>
-sudo gem install mysql2 #or postgres
+sudo gem install mysql2 #or sqlite3-ruby #or postgres
 </code>
 
 #### Configuring
@@ -95,12 +94,14 @@ and performance reasons. To catch up with Github's event stream, it is
 usually enough to run `ght-mirror-events` on one host. To collect all data
 pointed by each event, one instance of `ght-data-retrieval` is not enough.
 Both scripts employ throttling mechanisms to keep API usage whithin the
-limits imposed by Github (currently 5000 reqs/hr).
+limits imposed by Github (currently 60 reqs/hr/ip). If you want the full
+5000 reqs/hr/ip, you will have to provide your Github login details 
+in the `config.yaml` file.
 
 #### Data
 
 You can find torrents for retrieving data on the 
-[Available Torrents](https://github.com/gousiosg/github-mirror/wiki/Available-Torrents) page. You need two sets of data:
+[Available Torrents](https://ghtorrent.org/downloads.html) page. You need two sets of data:
 
 * Raw events: Github's [event stream](https://api.github.com/events). These
 are the roots for mirroring operations. The `ght-data-retrieval` crawler starts
@@ -108,11 +109,6 @@ from an event and goes deep into the rabbit hole.
 * SQL dumps+Linked data: Data dumps from the SQL database and the corresponding
 MongoDB entities.
 
-
-*At the moment, GHTorrent is in the process of redesigning its data storage
-schema. Consequently, it does not distribute SQL dumps or linked data raw data.
-The distribution service will come back shortly.*
-
 #### Reporting bugs
 
 Please use the [Issue

data/bin/ght-retrieve-user

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'ghtorrent'
+
+GHTRetrieveUser.run(ARGV)

data/lib/ghtorrent.rb

@@ -56,5 +56,6 @@ require 'ghtorrent/commands/ght_get_more_commits'
 require 'ghtorrent/commands/ght_rm_dupl'
 require 'ghtorrent/commands/ght_load'
 require 'ghtorrent/commands/ght_retrieve_repo'
+require 'ghtorrent/commands/ght_retrieve_user'
 
 # vim: set sta sts=2 shiftwidth=2 sw=2 et ai :

data/lib/ghtorrent/adapters/base_adapter.rb

@@ -70,6 +70,12 @@ module GHTorrent
       end
     end
 
+    def del(entity, query = {})
+      unless ENTITIES.include?(entity)
+        raise GHTorrentException.new("Perister: Entity #{entity} not known")
+      end
+    end
+
     # Get a raw connection to the underlying data store. The connection is
     # implementaiton dependent.
     def get_underlying_connection

data/lib/ghtorrent/adapters/mongo_persister.rb

@@ -65,6 +65,12 @@ module GHTorrent
       end
     end
 
+    def del(entity, query)
+      super
+      raise Exception 'No filter was specifed. Cowardily refusing to remove all entries' if query == {}
+      get_entity(entity).remove(query)
+    end
+
     def get_underlying_connection
       mongo
     end
@@ -163,7 +169,9 @@ module GHTorrent
       ensure_index(:users, "login")
       ensure_index(:commits, "sha")
       ensure_index(:repos, "name")
+      ensure_index(:repos, "owner.login")
       ensure_index(:followers, "follows")
+      ensure_index(:followers, "login")
       ensure_index(:org_members, "org")
       ensure_index(:commit_comments, "repo")
       ensure_index(:commit_comments, "user")

data/lib/ghtorrent/api_client.rb

@@ -25,7 +25,8 @@ module GHTorrent
 
     # A paged request. Used when the result can expand to more than one
     # result pages.
-    def paged_api_request(url, pages = -1, cache = true, last = nil)
+    def paged_api_request(url, pages = config(:mirror_history_pages_back),
+        cache = true, last = nil)
 
       url = if not url.include?("per_page")
               if url.include?("?")
@@ -139,24 +140,6 @@ module GHTorrent
 
     # Do the actual request and return the result object
     def api_request_raw(url, use_cache = false)
-      @num_api_calls ||= 0
-      @ts ||= Time.now.to_i
-      @started_min ||= Time.now.min
-
-      #Rate limiting to avoid error requests
-      if Time.now().tv_sec() - @ts < 60 then
-        if @num_api_calls >= @settings['mirror']['reqrate'].to_i
-          sleep = 60 - (Time.now.to_i - @ts)
-          debug "APIClient: Sleeping for #{sleep}"
-          sleep (sleep)
-          @num_api_calls = 0
-          @ts = Time.now.to_i
-        end
-      else
-        debug "APIClient: Tick, num_calls = #{@num_api_calls}, zeroing"
-        @num_api_calls = 0
-        @ts = Time.now.to_i
-      end
 
       begin
         start_time = Time.now
@@ -169,25 +152,21 @@ module GHTorrent
                 cached
               else
                 tocache = Cachable.new(do_request(url))
-                @num_api_calls += 1
                 cache_put(url, tocache)
                 tocache
               end
             else
-              @num_api_calls += 1
               do_request(url)
             end
 
         total = Time.now.to_ms - start_time.to_ms
-        debug "APIClient: Request: #{url} (#{@num_api_calls} calls #{if from_cache then " from cache," else "(#{contents.meta['x-ratelimit-remaining']} remaining)," end} Total: #{total} ms)"
+        debug "APIClient: Request: #{url} #{if from_cache then " from cache," else "(#{contents.meta['x-ratelimit-remaining']} remaining)," end} Total: #{total} ms"
 
         if not from_cache and config(:respect_api_ratelimit) and
-            contents.meta['x-ratelimit-remaining'].to_i < 400
-          sleep = 60 - @started_min
+            contents.meta['x-ratelimit-remaining'].to_i < 20
+          sleep = 61 - Time.now.min
           debug "APIClient: Request limit reached, sleeping for #{sleep} min"
           sleep(sleep * 60)
-          @started_min = Time.now.min
-          @num_api_calls = 0
         end
 
         contents
@@ -199,10 +178,10 @@ module GHTorrent
               403, # Forbidden
               404, # Not found
               422 then # Unprocessable entity
-            warn "#{url}: #{e.io.status[1]}"
+            warn "APIClient: #{url}: #{e.io.status[1]}"
             return nil
           else # Server error or HTTP conditions that Github does not report
-            warn "#{url}"
+            warn "APIClient: #{url}: #{e.io.status[1]}"
             raise e
         end
       end
@@ -212,7 +191,7 @@ module GHTorrent
       @attach_ip ||= config(:attach_ip)
       @username ||= config(:github_username)
       @passwd ||= config(:github_passwd)
-      @user_agent ||= "ghtorrent-v#{GHTorrent::VERSION}"
+      @user_agent ||= config(:user_agent)
 
       @open_func ||= if @username.nil?
         lambda {|url| open(url, 'User-Agent' => @user_agent)}

data/lib/ghtorrent/command.rb

@@ -35,8 +35,6 @@ module GHTorrent
         command.process_options
         command.validate
 
-        puts "GHTorrent version: #{GHTorrent::VERSION}"
-
         command.settings = YAML::load_file command.options[:config]
 
         unless command.options[:addr].nil?
@@ -165,7 +163,7 @@ Standard options:
     end
 
     def override_config(config_file, setting, new_value)
-      STDERR.puts "Overriding configuration #{setting}=#{config(setting)} with cmd line #{new_value}"
+      puts "Overriding configuration #{setting}=#{config(setting)} with cmd line #{new_value}"
       merge_config_values({setting => new_value})
     end
 

data/lib/ghtorrent/commands/ght_data_retrieval.rb

@@ -55,9 +55,8 @@ class GHTDataRetrieval < GHTorrent::Command
     user = data['actor']['login']
     repo = data['repo']['name'].split(/\//)[1]
     id = data['payload']['comment']['id']
-    created_at = data['created_at']
 
-    ghtorrent.get_commit_comment(user, repo, id, created_at)
+    ghtorrent.get_commit_comment(user, repo, id)
   end
 
   def PullRequestEvent(data)
@@ -74,9 +73,8 @@ class GHTDataRetrieval < GHTorrent::Command
     owner = data['repo']['name'].split(/\//)[0]
     repo = data['repo']['name'].split(/\//)[1]
     fork_id = data['payload']['forkee']['id']
-    created_at = data['created_at']
 
-    ghtorrent.get_fork(owner, repo, fork_id, created_at)
+    ghtorrent.get_fork(owner, repo, fork_id)
   end
 
   def PullRequestReviewCommentEvent(data)
@@ -84,18 +82,16 @@ class GHTDataRetrieval < GHTorrent::Command
     repo = data['repo']['name'].split(/\//)[1]
     comment_id = data['payload']['comment']['id']
     pullreq_id = data['payload']['comment']['_links']['pull_request']['href'].split(/\//)[-1]
-    created_at = data['created_at']
 
-    ghtorrent.get_pullreq_comment(owner, repo, pullreq_id, comment_id, created_at)
+    ghtorrent.get_pullreq_comment(owner, repo, pullreq_id, comment_id)
   end
 
   def IssuesEvent(data)
     owner = data['repo']['name'].split(/\//)[0]
     repo = data['repo']['name'].split(/\//)[1]
     issue_id = data['payload']['issue']['number']
-    created_at = data['created_at']
 
-    ghtorrent.get_issue(owner, repo, issue_id, created_at)
+    ghtorrent.get_issue(owner, repo, issue_id)
   end
 
   def IssueCommentEvent(data)
@@ -103,7 +99,6 @@ class GHTDataRetrieval < GHTorrent::Command
     repo = data['repo']['name'].split(/\//)[1]
     issue_id = data['payload']['issue']['number']
     comment_id = data['payload']['comment']['id']
-    created_at = data['created_at']
 
     ghtorrent.get_issue_comment(owner, repo, issue_id, comment_id)
   end
@@ -112,7 +107,7 @@ class GHTDataRetrieval < GHTorrent::Command
     %w(PushEvent WatchEvent FollowEvent MemberEvent
         CommitCommentEvent PullRequestEvent ForkEvent
         PullRequestReviewCommentEvent IssuesEvent IssueCommentEvent)
-    #%w(IssuesEvent IssueCommentEvent)
+    #%w(PullRequestEvent)
   end
 
   def prepare_options(options)

data/lib/ghtorrent/commands/ght_get_more_commits.rb

@@ -23,17 +23,17 @@ Retrieves more commits for the provided repository
     BANNER
 
     options.opt :num, 'Number of commits to retrieve',
-                :short => 'n', :default => -1, :type => :int
-    options.opt :full, 'Retrieve all commits, filling in potential holes',
-                :short => 'f', :default => -1, :type => :int
+                :short => 'n', :default => 1024 * 1024 * 1024, :type => :int
+    options.opt :full, 'Retrieve all commits, starting from the latest available.
+                        If not set, will start from latest stored commit',
+                :short => 'f', :default => false, :type => :boolean
+    options.opt :upto, 'Get all commits up to the provided timestamp',
+                :short => 't', :default => 0, :type => :int
   end
 
   def validate
     super
     Trollop::die "Two arguments are required" unless args[0] && !args[0].empty?
-
-    Trollop::die "-a and -n cannot be defined at the same time" \
-      if not options[:all].nil? and not options[:foo].nil?
   end
 
   def logger
@@ -68,37 +68,43 @@ Retrieves more commits for the provided repository
     end
 
     repo = repo_entry[:name]
-    num_pages = if options[:num] == -1 then 1024 * 1024 else options[:n]/30 end
-    num_pages = if options[:full] == -1 then num_pages else 1024 * 1024 end
-    page = 0
-
 
-    head = unless options[:full] == -1
+    head = if options[:full] == false
              @ght.get_db.from(:commits).\
                       where(:commits__project_id => repo_entry[:id]).\
                       order(:created_at).\
-                      first.\
-                      select(:sha)
+                      first[:sha]
            else
              "master"
            end
 
     total_commits = 0
-    while (page < num_pages)
+    old_head = nil
+    while (true)
       begin
         logger.debug("Retrieving more commits for #{user}/#{repo} from head: #{head}")
 
         commits = retrieve_commits(repo, head, user, 1)
-        page += 1
+
         if commits.nil? or commits.empty? or commits.size == 1
-          page = num_pages # To break the loop
           break
         end
 
-        total_commits += commits.size
         head = commits.last['sha']
 
         commits.map do |c|
+          total_commits += 1
+
+          if options[:num] < total_commits
+            logger.info("Already retrieved #{total_commits} commits. Stopping.")
+            return
+          end
+
+          if Time.parse(c['commit']['author']['date']) < Time.at(options[:upto])
+            logger.info("Commit #{c['sha']} older than #{Time.at(options[:upto])}. Stopping.")
+            return
+          end
+
           @ght.transaction do
             @ght.ensure_commit(repo, c['sha'], user)
           end
@@ -106,6 +112,11 @@ Retrieves more commits for the provided repository
       rescue Exception => e
         logger.warn("Error processing: #{e}")
         logger.warn(e.backtrace.join("\n"))
+        if old_head == head
+          logger.info("Commit #{c['sha']} older than #{Time.at(options[:upto])}. Stopping.")
+          fail("Cannot retrieve commits from head: #{head}")
+        end
+        old_head = head
       end
     end
     logger.debug("Processed #{total_commits} commits for #{user}/#{repo}")