ghtorrent 0.10 → 0.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1394c5a3869bab8d02cf61fc8108ca1f10afc0b8
-  data.tar.gz: 9d82ecf68d0316db32f962af8cb7d0070dd74c82
+  metadata.gz: 68b11841a8dfbd0418723fce0620a5d625b4cca1
+  data.tar.gz: d39f30596d257cfe5cb365e1365388169e50cc18
 SHA512:
-  metadata.gz: 7dbfe453c542137cafafa641c83bb680749e946c86193447e2b6aee829b88370bc66e90a8ea3d64a33d3c0061a7c5b01fa292ee96b94fba6dcca910d472d0f7f
-  data.tar.gz: d5c6487264b3a4d17af08e33ac2babd17b63ca6849e1ebd694a1d6b639797e8e870664f7dcde79718d61350551971a0c4b99b5d0ec56923b31984227c69b49bc
+  metadata.gz: 6443373ff38703c8113c23716db591a1574eb7f2e36eba1ae68d78ce322b20e42f1c0c840c3a113bb870b7c8b5aae817cb07a60f46d6bccb07ed4058cabc23d2
+  data.tar.gz: cfad88e464fad602f38f6f4cfb7963a0b7ac28f5c205ddf954939937c398d513e004725ef70e920b682b06239c2adc6663c3a29f6eea90913aad3611c9c2a310

data/CHANGELOG

@@ -1,3 +1,19 @@
+= Version 0.11
+	* Retrieve members by processing MemberEvents to counter API change
+	* Removed the request caching layer. Requests are cached in the persister
+	* Change default DB isolation to REPEATABLE READ for stronger isolation
+	* Finer-grained (commit level) transactions when processing forks
+    * More accurate and uniform logging
+    * Tool to push logs to InfluxDB for monitoring
+    * Drop ext_ref_id from all tables
+    * More efficient retrieval of events, 100 in one go
+    * Tool to retrieve all user details and support for marking users deleted
+    * Support for retrieving repo events when using ght-retrieve-repo
+    * Non-recursive retrieval of pull requests leads to 1/3 API calls
+    * Custom rate limits for Github API tokens
+    * Tooling for MySQL dumps in CSV files
+    * General bug fixes and cleanups
+
 = Version 0.10
 	* Base class for multiprocess queue clients
 	* Make retrieval of pull request commits faster

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ghtorrent (0.9)
+    ghtorrent (0.11)
       bson_ext (~> 1.9, >= 1.9.0)
       bunny (~> 1.0, >= 1.0.0)
       mongo (~> 1.9, >= 1.9.0)
@@ -11,32 +11,16 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    addressable (2.3.5)
     amq-protocol (1.9.2)
-    bson (1.10.0)
-    bson_ext (1.10.0)
-      bson (~> 1.10.0)
-    bunny (1.2.1)
+    bson (1.12.2)
+    bson_ext (1.12.2)
+      bson (~> 1.12.2)
+    bunny (1.3.1)
       amq-protocol (>= 1.9.2)
-    crack (0.4.1)
-      safe_yaml (~> 0.9.0)
-    diff-lcs (1.2.5)
-    mongo (1.10.0)
-      bson (~> 1.10.0)
-    rspec (2.14.1)
-      rspec-core (~> 2.14.0)
-      rspec-expectations (~> 2.14.0)
-      rspec-mocks (~> 2.14.0)
-    rspec-core (2.14.7)
-    rspec-expectations (2.14.4)
-      diff-lcs (>= 1.1.3, < 2.0)
-    rspec-mocks (2.14.4)
-    safe_yaml (0.9.7)
-    sequel (4.10.0)
-    trollop (2.0)
-    webmock (1.16.0)
-      addressable (>= 2.2.7)
-      crack (>= 0.3.2)
+    mongo (1.12.2)
+      bson (= 1.12.2)
+    sequel (4.23.0)
+    trollop (2.1.2)
 
 PLATFORMS
   ruby
@@ -44,5 +28,6 @@ PLATFORMS
 DEPENDENCIES
   ghtorrent!
   jdbc-mysql
-  rspec (~> 2.14, >= 2.14.0)
-  webmock (~> 1.16)
+
+BUNDLED WITH
+   1.10.2

data/README.md

@@ -1,4 +1,4 @@
-# ghtorrent: Mirror and process data from the Github API 
+# ghtorrent: Mirror and index data from the Github API
 
 A library and a collection of scripts used to retrieve data from the Github API
 and extract metadata in an SQL database, in a modular and scalable manner. The
@@ -10,8 +10,7 @@ GHTorrent can be used for a variety of purposes, such as:
 * Mirror the Github API event stream and follow links from events to actual data
  to gradually build a [Github index](http://ghtorrent.org/)
 * Create a queriable metadata index for a specific repository
-* Query the Github API using intelligent caching to avoid duplicate queries
-
+* Construct a data source for [extracting process analytics](http://www.gousios.gr/blog/ghtorrent-project-statistics/) (see for example [those](http://ghtorrent.org/pullreq-perf/))for one or more repositories
 
 ## Components
 
@@ -19,8 +18,8 @@ GHTorrents components (which can be used individually) are:
 
 * [APIClient](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/api_client.rb): Knows how to query the Github API (both single entities and
 pages) and respect the API request limit. Can be configured to override the
-default IP address, in case of multihomed hosts. Uses configurable on disk [caching](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/cache.rb) to avoid retrieving data that do not change.
-* [Retriever](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/retriever.rb): Knows how to retrieve specific Github entities (users, repositories, watchers) by name. Uses an optional persister to avoid 
+default IP address, in case of multihomed hosts.
+* [Retriever](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/retriever.rb): Knows how to retrieve specific Github entities (users, repositories, watchers) by name. Uses an optional persister to avoid
 retrieving data that have not changed.
 * [Persister](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/persister.rb): A key/value store, which can be backed by a real key/value store,
 to store Github JSON replies and query them on request. The backing key/value
@@ -28,25 +27,23 @@ store must support arbitrary queries to the stored JSON objects.
 * [GHTorrent](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/ghtorrent.rb): Knows how to extract information from the data retrieved by
 the retriever in order to update an SQL database (see [schema](http://ghtorrent.org/relational.html)) with metadata.
 
-### Component Configuration 
+### Component Configuration
 
 The Persister and GHTorrent components have configurable back ends:
 
 * **Persister:** Either uses MongoDB > 2.0 (`mongo` driver) or no persister (`noop` driver)
-* **GHTorrent:** GHTorrent is tested mainly with MySQL, but can theoretically be
-used with any SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). Your milaege may vary.
-
-The distributed mirroring scripts also require RabbitMQ >= 2.8 or other
+* **GHTorrent:** GHTorrent is tested mainly with MySQL and SQLite, but can theoretically be used with any SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). Your milaege may vary.
 
+For distributed mirroring you also need RabbitMQ >= 3
 
 ## Installation
 
 
 ### 1. Install GHTorrent
-GHTorrent is written in Ruby (tested with 1.9). To install it as a Gem do:
+GHTorrent is written in Ruby (tested with 2.0). To install it as a Gem do:
 
 <code>
-sudo gem install ghtorrent 
+sudo gem install ghtorrent
 </code>
 
 
@@ -56,14 +53,14 @@ Depending on which SQL database you want to use, install the appropriate
 dependency gem.
 
 <code>
-sudo gem install mysql2 # or <sqlite3-ruby|postgres>
+sudo gem install mysql2 # or sqlite3
 </code>
 
 
 ## Configuration
 
 Copy [config.yaml.tmpl](https://github.com/gousiosg/github-mirror/blob/master/config.yaml.tmpl)
-to a file in your home directory. 
+to a file in your home directory.
 
 All provided scripts accept the `-c` option, which accepts the location of the configuration file as
 a parameter.
@@ -74,7 +71,7 @@ to retrieve data in parallel on the [Wiki](https://github.com/gousiosg/github-mi
 
 ## Using GHTorrent
 
-To mirror the event stream and capture all data: 
+To mirror the event stream and capture all data:
 
 * `ght-mirror-events.rb` periodically polls Github's event
 queue (`https://api.github.com/events`), stores all new events in the
@@ -85,7 +82,7 @@ RabbitMQ.
 functions. The functions use the appropriate Github API call to retrieve the
 linked contents, extract metadata (for database storage), and store the
 retrieved data in the appropriate collection in the persister, to avoid
-duplicate API calls. 
+duplicate API calls.
 Data in the SQL database contain pointers (the `ext_ref_id` field) to the
 "raw" data in the persister.
 
@@ -98,32 +95,29 @@ To perform maintenance:
 
 * `ght-load` loads selected events from the persister to the queue in order for
 the `ght-data-retrieval` script to reprocess them
-* `ght-get-more-commits` retrieves all commits for a specific repository
-
 
-### Data Torrents
+### Data
 
-You can find torrents for retrieving data on the 
-[Available Torrents](https://ghtorrent.org/downloads.html) page. 
+The code in this repository is used to power the data collection process of
+the [GHTorrent.org](http://ghtorrent.org/) project.
+You can find all data collected by in the project in the
+[Downloads](https://ghtorrent.org/downloads.html) page.
 
 There are two sets of data:
 
 * **Raw events:** Github's [event stream](https://api.github.com/events). These
 are the roots for mirroring operations. The `ght-data-retrieval` crawler starts
 from an event and goes deep into the rabbit hole.
-* **SQL dumps + Linked data:** Data dumps from the SQL database and the corresponding
-MongoDB entities.
-
+* **SQL dumps + Linked data:** Data dumps from the SQL database and the corresponding MongoDB entities.
 
 ## Bugs & Feature Requests
 
-Please tell us about features you'd like or bugs you've discovered on our 
+Please tell us about features you'd like or bugs you've discovered on our
 [Issue Tracker](https://github.com/gousiosg/github-mirror/issues).
 
 Patches, bug fixes, etc are welcome. Please fork the repository and create
 a pull request when done fixing/implementing the new feature.
 
-
 ## Citing GHTorrent in your Research
 
 If you find GHTorrent and the accompanying datasets useful in your research,
@@ -131,18 +125,11 @@ please consider citing the following paper:
 
 > Georgios Gousios and Diomidis Spinellis, "GHTorrent: GitHub’s data from a firehose," in _MSR '12: Proceedings of the 9th Working Conference on Mining Software Repositories_, June 2-–3, 2012. Zurich, Switzerland.
 
-See also the following presentation:
-
-<iframe src="http://www.slideshare.net/slideshow/embed_code/13184524?rel=0" width="342" height="291" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC;border-width:1px 1px 0;margin-bottom:5px" allowfullscreen /> 
-<div style="margin-bottom:5px"> <strong> <a href="http://www.slideshare.net/gousiosg/ghtorrent-githubs-data-from-a-firehose-13184524" title="GHTorrent: Github&#39;s Data from a Firehose" target="_blank">GHTorrent: Github&#39;s Data from a Firehose</a> </strong> </div>
-
-
 ## Authors
 
 * [Georgios Gousios](http://istlab.dmst.aueb.gr/~george) <gousiosg@gmail.com>
 * [Diomidis Spinellis](http://www.dmst.aueb.gr/dds) <dds@aueb.gr>
 
-
 ## License
 
 [2-clause BSD](http://www.opensource.org/licenses/bsd-license.php)

data/Rakefile

@@ -1,15 +1,7 @@
 require 'rake'
-require 'rake/testtask'
 require 'rake/rdoctask'
 
-task :default => [:spec, :rdoc]
-
-desc "Run basic tests"
-Rake::TestTask.new(:spec) do |t|
-  t.pattern = 'spec/*_test.rb'
-  t.verbose = true
-  t.warning = true
-end
+task :default => [:rdoc]
 
 desc "Run Rdoc"
 Rake::RDocTask.new(:rdoc) do |rd|

data/bin/ght-log-analyzer

@@ -9,7 +9,7 @@ Thread.new do
   puts "Collecting data..."
   while (true) do
     sleep(1)
-    system "clear" or system "cls"
+    system 'clear' or system 'cls'
 
     stats.each do |k,v|
       unless v[:time_in].nil?
@@ -68,7 +68,7 @@ end
 
 ARGF.each do |x|
 
-  next unless x =~ /APIClient/
+  next unless x =~ /api_client.rb/
 
   if x =~ /sleeping/
     ts, pid, remaining = x.match(/\[([^.]+).*#([0-9]+)\].*for ([0-9]+).*/).captures
@@ -82,15 +82,20 @@ ARGF.each do |x|
     end
 
   elsif x =~ /Not Found|Gone|Conflict/
-    pid = x.match(/.*#([0-9]+).*APIClient.*/).captures[0]
+    pid = x.match(/.*#([0-9]+).*api_client.rb.*/).captures[0]
     if stats[pid][:not_found].nil?
       stats[pid][:not_found] = 0
-  end
-  stats[pid][:not_found] += 1
+    end
+    stats[pid][:not_found] += 1
+  elsif x =~ /Forbidden/
+    if stats[pid][:forbidden].nil?
+      stats[pid][:forbidden] = 0
+    end
+    stats[pid][:forbidden] += 1
   else
     begin
       ts, pid, ip, url, remaining, time =
-        x.match(/.*\[([^.]+).*#([0-9]+)\].*APIClient\[(.*)\].*(https:\/\/.*) \(([0-9]+) remaining\).* ([0-9]+) ms$/).captures
+        x.match(/.*\[([^.]+).*#([0-9]+)\].*api_client.rb: \[(.*)\].*(https:\/\/.*) \(([0-9]+) remaining\).* ([0-9]+) ms$/).captures
     rescue
       puts x
       next

data/bin/ght-log-influx

@@ -0,0 +1,190 @@
+#!/usr/bin/env ruby
+#
+
+require 'rubygems'
+require 'trollop'
+require 'influxdb'
+require 'pp'
+require 'time'
+
+def parse_api_client_line(line)
+  if line.start_with?("Successful")
+    # Successful request. URL: https://api.github.com/repos/amizony/self-destructing-task-list/comments/11518274?per_page=100, Remaining: 3949, Total: 423 ms
+    remaining, elapsed = line.match(/.*Remaining: ([\d]+), Total: ([\d]+) ms/).captures
+    {
+        :outcome => "success",
+        :elapsed => elapsed.to_i,
+        :remaining => remaining.to_i
+    }
+  elsif line.start_with?("Failed")
+    # Failed request. URL: https://api.github.com/repos/mingliang7/hotel/commits?per_page=100, Status code: 409, Status: Conflict, Access: ghtorrent, IP: 0.0.0.0, Remaining: 3332
+    code, elapsed = line.match(/.*Status code: ([^,]+), .*Remaining: ([\d]+)/).captures
+    {
+        :outcome => "error",
+        :error_code => code.to_i,
+        :remaining => remaining.to_i
+    }
+  else
+    {}
+  end
+end
+
+def parse_data_retrieval_line(line)
+  #Success processing event. Type: PushEvent, ID: 2863181313, Time: 967 ms
+  return {} unless line.start_with?("Success") or line.start_with?("Error")
+  outcome, evt_type, time = line.match(/([^\ ]+) processing event\. Type: ([\D]+)Event, .*, Time: ([\d]+) ms/).captures
+
+  {
+      :outcome => outcome.downcase,
+      :evt_type => evt_type,
+      :elapsed => time.to_i
+  }
+end
+
+def parse_retriever_line(line)
+
+  if line.start_with?("Added")
+    # Added repo hiropong -> googlemaplesson
+    outcome = "success"
+    entity = line.split(/ /)[1]
+  elsif line.start_with?("Could not find")
+    # Could not find commit_comment 12106552. Deleted?
+    outcome = "failure"
+    entity = line.split(/ /)[3]
+  else
+    return {}
+  end
+
+  {
+      :outcome => outcome,
+      :entity => entity
+  }
+
+end
+
+def parse_ghtorrent_line(line)
+
+  if line.start_with?("Added")
+    # Added user hayjohnny2000
+    # Added issue_event etsy/logster -> 1/etsy/logster -> 1/417355
+    outcome = "success"
+    entity = line.split(/ /)[1]
+  elsif line.start_with?("Could not retrieve")
+    # Could not retrieve commit_comment 12106552. Deleted?
+    outcome = "failure"
+    entity = line.split(/ /)[3]
+  else
+    return {}
+  end
+
+  {
+      :outcome => outcome,
+      :entity => entity
+  }
+
+end
+
+def parse_log_line(line)
+  begin
+    severity, time, progname, stage, msg =
+        line.match(/([A-Z]+), (.+), (.+) -- ([^:]*?): (.*)/).captures
+  rescue
+    puts "Error parsing line: #{line}"
+    return {}
+  end
+
+  return {} if severity.downcase == 'debug'
+  stage = stage.split(/\./)[0]
+  data = {
+      :time => Time.iso8601(time).to_f,
+      :client => progname,
+      :severity => severity,
+      :stage => stage
+  }
+
+  return {} if msg.nil? or msg.length == 0
+
+
+  stage_specific =
+    begin
+      case stage
+        when 'api_client'
+          parse_api_client_line(msg)
+        when 'ght_data_retrieval'
+          parse_data_retrieval_line(msg)
+        when 'retriever'
+          parse_retriever_line(msg)
+        when 'ghtorrent'
+          parse_ghtorrent_line(msg)
+        else
+          {}
+      end
+    rescue
+      puts "Error parsing line: #{msg}"
+      {}
+    end
+
+  return {} if stage_specific.empty?
+  data.merge(stage_specific)
+end
+
+opts = Trollop::options do
+  banner <<-END
+  Store GHTorrent log output to InfluxDB. By default reads from STDIN.
+  Can be configured to watch files in directories.
+
+Options:
+  END
+
+  opt :watch, "Use watch mode", :sort => 'w'
+  opt :watch_pattern, "Pattern for files to watch",
+    :short => 'p', :default => '*.log'
+
+  opt :db_server, "InfluxDB server to use", :type => String,
+    :short => 's', :default => '127.0.0.1'
+  opt :database, "InfluxDB database to use", :type => String,
+      :short => 'd', :default => 'ghtorrent'
+  opt :db_uname, "Username for the Influx database", :type => String,
+      :short => 'u', :default => 'ghtorrent'
+  opt :db_passwd, "Password for the Influx database", :type => String,
+      :short => 'x', :default => ''
+end
+
+unless opts[:db_passwd_given].nil?
+  influx = InfluxDB::Client.new(opts[:database],
+                                :host     => opts[:db_server],
+                                :username => opts[:db_uname],
+                                :password => opts[:db_passwd])
+else
+  influx = InfluxDB::Client.new("ghtorrent",
+                                :host => opts[:db_server])
+end
+influx.get_database_list
+
+if opts[:watch]
+  require 'filewatch/tail'
+
+  t = FileWatch::Tail.new
+  t.tail(opts[:watch_pattern])
+
+  t.subscribe do |path, line|
+    p = parse_log_line(line)
+    next if p.empty?
+
+    pp p
+    #influx.write_point(p[:stage], p)
+  end
+else
+  puts "Reading from STDIN..."
+  ARGF.each do |line|
+    next if line !~ /^[IDEW]/
+
+    begin
+      p = parse_log_line(line)
+      next if p.empty?
+      pp p
+      influx.write_point(p[:stage], p)
+    rescue
+    end
+  end
+end