ghtorrent 0.7.1 → 0.7.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/CHANGELOG +3 -0
- data/Gemfile.lock +2 -2
- data/README.md +55 -64
- data/lib/ghtorrent/adapters/noop_persister.rb +1 -1
- data/lib/ghtorrent/api_client.rb +20 -11
- data/lib/ghtorrent/cache.rb +3 -3
- data/lib/ghtorrent/ghtorrent.rb +2 -1
- data/lib/ghtorrent/migrations/012_add_forks_to_projects.rb +8 -7
- data/lib/ghtorrent/migrations/013_add_merged_to_pullreqs.rb +17 -15
- data/lib/ghtorrent/persister.rb +2 -2
- data/lib/ghtorrent/retriever.rb +9 -3
- data/lib/ghtorrent/settings.rb +44 -39
- data/lib/version.rb +1 -1
- metadata +4 -20
data/CHANGELOG
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
ghtorrent (0.
|
4
|
+
ghtorrent (0.7.2)
|
5
5
|
amqp (~> 1.0.0)
|
6
6
|
bson_ext (~> 1.8.0)
|
7
7
|
daemons (~> 1.1.0)
|
@@ -15,7 +15,7 @@ GEM
|
|
15
15
|
amq-client (1.0.2)
|
16
16
|
amq-protocol (>= 1.2.0)
|
17
17
|
eventmachine
|
18
|
-
amq-protocol (1.
|
18
|
+
amq-protocol (1.5.0)
|
19
19
|
amqp (1.0.2)
|
20
20
|
amq-client (~> 1.0.2)
|
21
21
|
amq-protocol (>= 1.3.0)
|
data/README.md
CHANGED
@@ -1,16 +1,41 @@
|
|
1
|
-
ghtorrent: Mirror and process the Github
|
2
|
-
|
1
|
+
##ghtorrent: Mirror and process data from the Github API
|
2
|
+
|
3
|
+
A library and a collection of scripts used to retrieve data from the Github API
|
4
|
+
and extract metadata in an SQL database, in a modular and scalable manner. The
|
5
|
+
scripts are distributed as a Gem (`ghtorrent`), but they can also be run by
|
6
|
+
checking out this repository.
|
7
|
+
|
8
|
+
GHTorrent can be used for a variety of purposes, such as:
|
9
|
+
|
10
|
+
* Mirror the Github API event stream and follow links from events to actual data
|
11
|
+
to gradually build a [Github index](http://ghtorrent.org/dblite/)
|
12
|
+
* Create a queriable metadata index for a specific repository
|
13
|
+
* Query the Github API using intelligent caching to avoid duplicate queries
|
14
|
+
|
15
|
+
GHTorrent is comprised from the following components (which can be used
|
16
|
+
individually):
|
17
|
+
|
18
|
+
* [APIClient](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/api_client.rb): Knows how to query the Github API (both single entities and
|
19
|
+
pages) and respect the API request limit. Can be configured to override the
|
20
|
+
default IP address, in case of multihomed hosts. Uses configurable on disk [caching](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/cache.rb) to avoid retrieving data that do not change.
|
21
|
+
* [Retriever](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/retriever.rb): Knows how to retrieve specific Github entities (users, repositories, watchers) by name. Uses an optional persister to avoid
|
22
|
+
retrieving data that have not changed.
|
23
|
+
* [Persister](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/persister.rb): A key/value store, which can be backed by a real key/value store,
|
24
|
+
to store Github JSON replies and query them on request. The backing key/value
|
25
|
+
store must support arbitrary queries to the stored JSON objects.
|
26
|
+
* [GHTorrent](https://github.com/gousiosg/github-mirror/blob/master/lib/ghtorrent/ghtorrent.rb): Knows how to extract information from the data retrieved by
|
27
|
+
the retriever in order to update an SQL database (see [schema](http://ghtorrent.org/relational.html)) with metadata.
|
28
|
+
|
29
|
+
The Persister and GHTorrent components have configurable back ends:
|
30
|
+
|
31
|
+
* Persister: Either uses MongoDB > 2.0 (`mongo` driver) or no persister (`noop` driver)
|
32
|
+
* GHTorrent: GHTorrent is tested mainly with MySQL, but can theoretically be
|
33
|
+
used with any SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). Your milaege may vary.
|
3
34
|
|
4
|
-
A collection of scripts used to mirror the Github event stream, for
|
5
|
-
research purposes. The scripts are distributed as a Gem (`ghtorrent`),
|
6
|
-
but they can also be run by checking out this repository.
|
7
35
|
|
8
|
-
|
36
|
+
The distributed mirroring scripts also require RabbitMQ >= 2.8 or other
|
9
37
|
|
10
|
-
|
11
|
-
* RabbitMQ >= 2.7
|
12
|
-
* MySQL >= 5.5. GHTorrent is tested mainly with MySQL, but can theoretically be
|
13
|
-
used with any SQL database compatible with [Sequel](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html). Your milaege may vary.
|
38
|
+
#### Installing
|
14
39
|
|
15
40
|
GHTorrent is written in Ruby (tested with 1.9). To install it as a Gem do:
|
16
41
|
|
@@ -19,8 +44,7 @@ sudo gem install ghtorrent
|
|
19
44
|
</code>
|
20
45
|
|
21
46
|
Depending on which SQL database you want to use, install the appropriate
|
22
|
-
dependency gem.
|
23
|
-
install the development package for `libmysql-dev` for your system).
|
47
|
+
dependency gem.
|
24
48
|
|
25
49
|
<code>
|
26
50
|
sudo gem install mysql2 #or sqlite3-ruby #or postgres
|
@@ -34,74 +58,41 @@ file to a file in your home directory. All provided scripts accept the `-c`
|
|
34
58
|
option, which you can use to pass the location of the configuration file as
|
35
59
|
a parameter.
|
36
60
|
|
37
|
-
|
38
|
-
require accounts with permissions to create queues and exchanges in the AMQP
|
39
|
-
queue, collections in MongoDB and tables in the selected SQL database,
|
40
|
-
respectively.
|
41
|
-
|
42
|
-
To prepare MongoDB:
|
43
|
-
|
44
|
-
<pre>
|
45
|
-
$ mongo admin
|
46
|
-
> db.addUser('github', 'github')
|
47
|
-
> use github
|
48
|
-
> db.addUser('github', 'github')
|
49
|
-
</pre>
|
50
|
-
|
51
|
-
To prepare RabbitMQ:
|
52
|
-
|
53
|
-
<pre>
|
54
|
-
$ rabbitmqctl add_user github
|
55
|
-
$ rabbitmqctl set_permissions -p / github ".*" ".*" ".*"
|
56
|
-
|
57
|
-
# The following will enable the RabbitMQ web admin for the github user
|
58
|
-
# Not necessary to have, but good to debug and diagnose problems
|
59
|
-
$ rabbitmq-plugins enable rabbitmq_management
|
60
|
-
$ rabbitmqctl set_user_tags github administrator
|
61
|
-
</pre>
|
62
|
-
|
63
|
-
To prepare MySQL:
|
64
|
-
|
65
|
-
<pre>
|
66
|
-
$ mysql -u root -p
|
67
|
-
mysql> create user 'github'@'localhost' identified by 'github';
|
68
|
-
mysql> create database github;
|
69
|
-
mysql> GRANT ALL PRIVILEGES ON github.* to github@'localhost';
|
70
|
-
mysql> flush privileges;
|
71
|
-
</pre>
|
72
|
-
|
73
|
-
You can find more information of how you can setup a cluster of machines
|
61
|
+
You can find more information of how you can setup a mirroring cluster of machines
|
74
62
|
to retrieve data in parallel on the [Wiki](https://github.com/gousiosg/github-mirror/wiki/Setting-up-a-mirroring-cluster).
|
75
63
|
|
76
64
|
### Running
|
77
65
|
|
78
|
-
To
|
66
|
+
To mirror the event stream and capture all data:
|
79
67
|
|
80
68
|
* `ght-mirror-events.rb` periodically polls Github's event
|
81
69
|
queue (`https://api.github.com/events`), stores all new events in the
|
82
|
-
|
70
|
+
configured pestister and posts them to the `github` exchange in
|
83
71
|
RabbitMQ.
|
84
72
|
|
85
73
|
* `ght-data_retrieval.rb` creates queues that route posted events to processor
|
86
74
|
functions, which in turn use the appropriate Github API call to retrieve the
|
87
75
|
linked contents, extract metadata to store in the SQL database and store the
|
88
|
-
retrieved data in the appropriate collection in
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
76
|
+
retrieved data in the appropriate collection in the persister, to avoid
|
77
|
+
duplicate API
|
78
|
+
calls. Data in the SQL database contain pointers (the `ext_ref_id` field) to the
|
79
|
+
"raw" data in the persister.
|
80
|
+
|
81
|
+
To retrieve data for a repository or user:
|
82
|
+
|
83
|
+
* `ght-retrieve-repo` retrieves all data for a specific repository
|
84
|
+
* `ght-retrieve-user` retrieves all data for a specific user
|
85
|
+
|
86
|
+
To perform maintenance:
|
87
|
+
|
88
|
+
* `ght-load` loads selected events from the persister to the queue in order for
|
89
|
+
the `ght-data-retrieval` script to reprocess them
|
90
|
+
* `ght-get-more-commits` retrieves all commits for a specific repository
|
100
91
|
|
101
92
|
#### Data
|
102
93
|
|
103
94
|
You can find torrents for retrieving data on the
|
104
|
-
[Available Torrents](https://ghtorrent.org/downloads.html) page. You
|
95
|
+
[Available Torrents](https://ghtorrent.org/downloads.html) page. You can find two sets of data:
|
105
96
|
|
106
97
|
* Raw events: Github's [event stream](https://api.github.com/events). These
|
107
98
|
are the roots for mirroring operations. The `ght-data-retrieval` crawler starts
|
data/lib/ghtorrent/api_client.rb
CHANGED
@@ -28,20 +28,25 @@ module GHTorrent
|
|
28
28
|
def paged_api_request(url, pages = config(:mirror_history_pages_back),
|
29
29
|
cache = true, last = nil)
|
30
30
|
|
31
|
-
url = if not url.include?(
|
32
|
-
if url.include?(
|
33
|
-
url +
|
31
|
+
url = if not url.include?('per_page')
|
32
|
+
if url.include?('?')
|
33
|
+
url + '&per_page=100'
|
34
34
|
else
|
35
|
-
url +
|
35
|
+
url + '?per_page=100'
|
36
36
|
end
|
37
37
|
else
|
38
38
|
url
|
39
39
|
end
|
40
40
|
|
41
|
-
|
41
|
+
params = CGI::parse(URI::parse(url).query)
|
42
|
+
data = if params.has_key?('page') or (params.has_key?('last_sha'))
|
42
43
|
api_request_raw(url, use_cache?(cache, method = :paged))
|
43
44
|
else
|
44
|
-
|
45
|
+
if @cache_mode == :all
|
46
|
+
api_request_raw(url, true)
|
47
|
+
else
|
48
|
+
api_request_raw(url, false)
|
49
|
+
end
|
45
50
|
end
|
46
51
|
|
47
52
|
return [] if data.nil?
|
@@ -77,7 +82,7 @@ module GHTorrent
|
|
77
82
|
|
78
83
|
# A normal request. Returns a hash or an array of hashes representing the
|
79
84
|
# parsed JSON result.
|
80
|
-
def api_request(url, cache =
|
85
|
+
def api_request(url, cache = true)
|
81
86
|
parse_request_result api_request_raw(url, use_cache?(cache))
|
82
87
|
end
|
83
88
|
|
@@ -87,14 +92,18 @@ module GHTorrent
|
|
87
92
|
# request
|
88
93
|
def use_cache?(client_request, method = :non_paged)
|
89
94
|
@cache_mode ||= case config(:cache_mode)
|
90
|
-
when
|
95
|
+
when 'dev'
|
91
96
|
:dev
|
92
|
-
when
|
97
|
+
when 'prod'
|
93
98
|
:prod
|
99
|
+
when 'all'
|
100
|
+
:all
|
94
101
|
else
|
95
102
|
raise GHTorrentException.new("Don't know cache configuration #{@cache_mode}")
|
96
103
|
end
|
97
104
|
case @cache_mode
|
105
|
+
when :all
|
106
|
+
return true
|
98
107
|
when :dev
|
99
108
|
unless client_request
|
100
109
|
return false
|
@@ -160,7 +169,7 @@ module GHTorrent
|
|
160
169
|
end
|
161
170
|
|
162
171
|
total = Time.now.to_ms - start_time.to_ms
|
163
|
-
debug "APIClient: Request: #{url} #{if from_cache then "
|
172
|
+
debug "APIClient: Request: #{url} #{if from_cache then "from cache," else "(#{contents.meta['x-ratelimit-remaining']} remaining)," end} Total: #{total} ms"
|
164
173
|
|
165
174
|
if not from_cache and config(:respect_api_ratelimit) and
|
166
175
|
contents.meta['x-ratelimit-remaining'].to_i < 20
|
@@ -201,7 +210,7 @@ module GHTorrent
|
|
201
210
|
:http_basic_authentication => [@username, @passwd])}
|
202
211
|
end
|
203
212
|
|
204
|
-
if @attach_ip.nil? or @attach_ip.eql?
|
213
|
+
if @attach_ip.nil? or @attach_ip.eql? '0.0.0.0'
|
205
214
|
@open_func.call(url)
|
206
215
|
else
|
207
216
|
attach_to(@attach_ip) do
|
data/lib/ghtorrent/cache.rb
CHANGED
@@ -33,7 +33,7 @@ module GHTorrent
|
|
33
33
|
end
|
34
34
|
end
|
35
35
|
rescue
|
36
|
-
warn "Could not cache object #{file} for key #{key}"
|
36
|
+
warn "Cache: Could not cache object #{file} for key #{key}"
|
37
37
|
end
|
38
38
|
end
|
39
39
|
|
@@ -47,7 +47,7 @@ module GHTorrent
|
|
47
47
|
end
|
48
48
|
|
49
49
|
unless (Time.now() - File.mtime(file)) < max_life
|
50
|
-
debug "Cached object for key #{key} too old"
|
50
|
+
debug "Cache: Cached object for key #{key} too old"
|
51
51
|
return nil
|
52
52
|
end
|
53
53
|
|
@@ -58,7 +58,7 @@ module GHTorrent
|
|
58
58
|
end
|
59
59
|
end
|
60
60
|
rescue
|
61
|
-
warn "Could not read object from cache location #{file}"
|
61
|
+
warn "Cache: Could not read object from cache location #{file}"
|
62
62
|
File.delete(file)
|
63
63
|
end
|
64
64
|
end
|
data/lib/ghtorrent/ghtorrent.rb
CHANGED
@@ -20,6 +20,7 @@ module GHTorrent
|
|
20
20
|
@settings = settings
|
21
21
|
@ext_uniq = config(:uniq_id)
|
22
22
|
@logger = Logger.new(STDOUT)
|
23
|
+
debug "GHTorrent: Using cache dir #{config(:cache_dir)}"
|
23
24
|
end
|
24
25
|
|
25
26
|
# Get a connection to the database
|
@@ -40,7 +41,7 @@ module GHTorrent
|
|
40
41
|
end
|
41
42
|
|
42
43
|
def persister
|
43
|
-
@persister ||= connect(:
|
44
|
+
@persister ||= connect(config(:mirror_persister), @settings)
|
44
45
|
@persister
|
45
46
|
end
|
46
47
|
|
@@ -3,20 +3,21 @@ require 'sequel'
|
|
3
3
|
require 'ghtorrent/migrations/mysql_defaults'
|
4
4
|
|
5
5
|
Sequel.migration do
|
6
|
+
|
6
7
|
up do
|
7
8
|
|
8
|
-
puts
|
9
|
+
puts 'Adding column forked_from in table projects'
|
9
10
|
alter_table :projects do
|
10
11
|
add_foreign_key :forked_from, :projects, :null => true
|
11
12
|
end
|
12
13
|
|
13
|
-
puts
|
14
|
-
|
15
|
-
|
16
|
-
fork =
|
14
|
+
puts 'Migrating data from forks to project(forked_from)'
|
15
|
+
self.transaction(:rollback => :reraise, :isolation => :committed) do
|
16
|
+
self[:projects].each do |p|
|
17
|
+
fork = self[:forks].first(:forked_project_id => p[:id])
|
17
18
|
unless fork.nil?
|
18
|
-
source =
|
19
|
-
|
19
|
+
source = self[:projects].first(:id => fork[:forked_from_id])
|
20
|
+
self[:projects].filter(:id => p[:id]).update(:forked_from => source[:id])
|
20
21
|
puts "#{p[:owner_id]}/#{p[:name]} is forked from #{source[:owner_id]}/#{source[:name]}"
|
21
22
|
end
|
22
23
|
end
|
@@ -10,26 +10,28 @@ Sequel.migration do
|
|
10
10
|
:default => false
|
11
11
|
|
12
12
|
puts "Updating pull_requests.merged"
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
13
|
+
self.transaction(:rollback => :reraise, :isolation => :committed) do
|
14
|
+
self << "update pull_requests
|
15
|
+
set merged = '1'
|
16
|
+
where exists (
|
17
|
+
select *
|
18
|
+
from pull_request_commits prc, project_commits pc
|
19
|
+
where prc.commit_id = pc.commit_id
|
20
|
+
and prc.pull_request_id = pull_requests.id
|
21
|
+
and pc.project_id = pull_requests.base_repo_id
|
22
|
+
and pull_requests.base_repo_id <> pull_requests.head_repo_id);"
|
23
|
+
|
24
|
+
self << "update pull_requests
|
25
|
+
set merged = '1'
|
24
26
|
where exists(
|
25
27
|
select prh.created_at
|
26
28
|
from pull_request_history prh
|
27
|
-
where prh.action='merged' and prh.pull_request_id=
|
29
|
+
where prh.action='merged' and prh.pull_request_id=pull_requests.id)"
|
28
30
|
end
|
29
31
|
|
30
|
-
puts
|
31
|
-
|
32
|
-
|
32
|
+
puts 'Correcting intra_branch field'
|
33
|
+
self.transaction(:rollback => :reraise, :isolation => :committed) do
|
34
|
+
self << "update pull_requests set intra_branch = '1' where base_repo_id = head_repo_id"
|
33
35
|
end
|
34
36
|
end
|
35
37
|
|
data/lib/ghtorrent/persister.rb
CHANGED
@@ -13,9 +13,9 @@ module GHTorrent
|
|
13
13
|
|
14
14
|
# Factory method for retrieving persistence connections.
|
15
15
|
# The +settings+ argument is a fully parsed YAML document
|
16
|
-
# passed on to adapters. The available +adapter+ are
|
16
|
+
# passed on to adapters. The available +adapter+ are 'mongo' and 'noop'
|
17
17
|
def connect(adapter, settings)
|
18
|
-
driver = ADAPTERS[adapter]
|
18
|
+
driver = ADAPTERS[adapter.intern]
|
19
19
|
driver.new(settings)
|
20
20
|
end
|
21
21
|
|
data/lib/ghtorrent/retriever.rb
CHANGED
@@ -559,7 +559,7 @@ module GHTorrent
|
|
559
559
|
paged_api_request(ghurl urls)
|
560
560
|
end
|
561
561
|
|
562
|
-
items.
|
562
|
+
items = items.map do |x|
|
563
563
|
x['repo'] = repo
|
564
564
|
x['owner'] = user
|
565
565
|
|
@@ -589,12 +589,18 @@ module GHTorrent
|
|
589
589
|
debug "Retriever: #{entity} #{user}/#{repo} -> #{x[descriminator]} exists"
|
590
590
|
end
|
591
591
|
end
|
592
|
+
# If the persistence driver does not set an ext_ref_id key, set a dummy
|
593
|
+
# one here
|
594
|
+
unless x.has_key? ext_uniq
|
595
|
+
x[ext_uniq] = '0'
|
596
|
+
end
|
597
|
+
x
|
592
598
|
end
|
593
599
|
|
594
600
|
if item_id.nil?
|
595
|
-
|
601
|
+
items
|
596
602
|
else
|
597
|
-
|
603
|
+
[items.find{|x| x[descriminator] == item_id}]
|
598
604
|
end
|
599
605
|
end
|
600
606
|
|
data/lib/ghtorrent/settings.rb
CHANGED
@@ -9,68 +9,73 @@ module GHTorrent
|
|
9
9
|
include GHTorrent::Utils
|
10
10
|
|
11
11
|
CONFIGKEYS = {
|
12
|
-
:amqp_host =>
|
13
|
-
:amqp_port =>
|
14
|
-
:amqp_username =>
|
15
|
-
:amqp_password =>
|
16
|
-
:amqp_exchange =>
|
17
|
-
:amqp_prefetch =>
|
12
|
+
:amqp_host => 'amqp.host',
|
13
|
+
:amqp_port => 'amqp.port',
|
14
|
+
:amqp_username => 'amqp.username',
|
15
|
+
:amqp_password => 'amqp.password',
|
16
|
+
:amqp_exchange => 'amqp.exchange',
|
17
|
+
:amqp_prefetch => 'amqp.prefetch',
|
18
18
|
|
19
|
-
:sql_url =>
|
19
|
+
:sql_url => 'sql.url',
|
20
20
|
|
21
|
-
:mirror_urlbase =>
|
22
|
-
:mirror_pollevery =>
|
23
|
-
:mirror_persister =>
|
24
|
-
:mirror_commit_pages_new_repo =>
|
25
|
-
:mirror_history_pages_back =>
|
26
|
-
:uniq_id =>
|
27
|
-
:user_agent =>
|
21
|
+
:mirror_urlbase => 'mirror.urlbase',
|
22
|
+
:mirror_pollevery => 'mirror.pollevery',
|
23
|
+
:mirror_persister => 'mirror.persister',
|
24
|
+
:mirror_commit_pages_new_repo => 'mirror.commit_pages_new_repo',
|
25
|
+
:mirror_history_pages_back => 'mirror.history_pages_back',
|
26
|
+
:uniq_id => 'mirror.uniq_id',
|
27
|
+
:user_agent => 'mirror.user_agent',
|
28
28
|
|
29
|
-
:cache_mode =>
|
30
|
-
:cache_dir =>
|
31
|
-
:cache_stale_age =>
|
29
|
+
:cache_mode => 'mirror.cache_mode',
|
30
|
+
:cache_dir => 'mirror.cache_dir',
|
31
|
+
:cache_stale_age => 'mirror.cache_stale_age',
|
32
32
|
|
33
|
-
:github_username =>
|
34
|
-
:github_passwd =>
|
33
|
+
:github_username => 'mirror.username',
|
34
|
+
:github_passwd => 'mirror.passwd',
|
35
35
|
|
36
|
-
:respect_api_ratelimit =>
|
36
|
+
:respect_api_ratelimit => 'mirror.respect_api_ratelimit',
|
37
37
|
|
38
|
-
:attach_ip =>
|
38
|
+
:attach_ip => 'mirror.attach_ip'
|
39
39
|
}
|
40
40
|
|
41
41
|
DEFAULTS = {
|
42
|
-
:amqp_host =>
|
42
|
+
:amqp_host => 'localhost',
|
43
43
|
:amqp_port => 5672,
|
44
|
-
:amqp_username =>
|
45
|
-
:amqp_password =>
|
46
|
-
:amqp_exchange =>
|
44
|
+
:amqp_username => 'github',
|
45
|
+
:amqp_password => 'github',
|
46
|
+
:amqp_exchange => 'github',
|
47
47
|
:amqp_prefetch => 1,
|
48
48
|
|
49
|
-
:sql_url =>
|
49
|
+
:sql_url => 'sqlite://github.db',
|
50
50
|
|
51
|
-
:mirror_urlbase =>
|
52
|
-
:mirror_pollevery =>
|
53
|
-
:mirror_persister =>
|
51
|
+
:mirror_urlbase => 'https://api.github.com/',
|
52
|
+
:mirror_pollevery => 'mirror.pollevery',
|
53
|
+
:mirror_persister => 'noop',
|
54
54
|
:mirror_commit_pages_new_repo => 3,
|
55
55
|
:mirror_history_pages_back => 1,
|
56
|
-
:uniq_id =>
|
57
|
-
:user_agent =>
|
56
|
+
:uniq_id => 'ext_ref_id',
|
57
|
+
:user_agent => 'ghtorrent',
|
58
58
|
|
59
|
-
:cache_mode =>
|
60
|
-
:cache_dir => Dir::tmpdir + File::SEPARATOR +
|
59
|
+
:cache_mode => 'dev',
|
60
|
+
:cache_dir => Dir::tmpdir + File::SEPARATOR + 'ghtorrent',
|
61
61
|
:cache_stale_age => 604800,
|
62
62
|
|
63
|
-
:github_username =>
|
64
|
-
:github_passwd =>
|
63
|
+
:github_username => 'foo',
|
64
|
+
:github_passwd => 'bar',
|
65
65
|
|
66
|
-
:respect_api_ratelimit =>
|
66
|
+
:respect_api_ratelimit => 'true',
|
67
67
|
|
68
|
-
:attach_ip =>
|
68
|
+
:attach_ip => '0.0.0.0'
|
69
69
|
}
|
70
70
|
|
71
71
|
def config(key, use_default = true)
|
72
72
|
begin
|
73
|
-
read_value(settings, CONFIGKEYS[key])
|
73
|
+
a = read_value(settings, CONFIGKEYS[key])
|
74
|
+
if a.nil? && use_default
|
75
|
+
DEFAULTS[key]
|
76
|
+
else
|
77
|
+
a
|
78
|
+
end
|
74
79
|
rescue Exception => e
|
75
80
|
if use_default
|
76
81
|
DEFAULTS[key]
|
@@ -91,7 +96,7 @@ module GHTorrent
|
|
91
96
|
end
|
92
97
|
|
93
98
|
def settings
|
94
|
-
raise Exception.new(
|
99
|
+
raise Exception.new('Unimplemented')
|
95
100
|
end
|
96
101
|
|
97
102
|
end
|
data/lib/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: ghtorrent
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.7.
|
4
|
+
version: 0.7.2
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date: 2013-05-
|
13
|
+
date: 2013-05-12 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: amqp
|
@@ -180,24 +180,8 @@ files:
|
|
180
180
|
homepage: https://github.com/gousiosg/github-mirror
|
181
181
|
licenses: []
|
182
182
|
post_install_message: !binary |-
|
183
|
-
|
184
|
-
|
185
|
-
WzMybVZlcnNpb24gMC43G1swbV0gQ2xlYW5lZCB1cCByZXRyaWV2YWwgb2Yg
|
186
|
-
cHVsbCByZXF1ZXN0IGNvbW1pdHMKWxtbMzJtVmVyc2lvbiAwLjcbWzBtXSBD
|
187
|
-
bGVhbmVkIHVwIGFzc29jaWF0aW9uIG9mIGNvbW1pdHMgd2l0aCByZXBvc2l0
|
188
|
-
b3JpZXMuClsbWzMybVZlcnNpb24gMC43G1swbV0gUmVtb3ZlZCB0aGUgZm9y
|
189
|
-
a3MgdGFibGUuIEZvcmtzIGFyZSBub3cgdHJhY2tlZCBieSB0aGUgZm9ya2Vk
|
190
|
-
X2Zyb20gZmllbGQgaW4gcHJvamVjdHMKWxtbMzJtVmVyc2lvbiAwLjcbWzBt
|
191
|
-
XSBVc2UgR2l0aHViJ3MgSFRUUCBoZWFkZXJzIGZvciByZXF1ZXN0IHRocm90
|
192
|
-
dGxpbmcKWxtbMzJtVmVyc2lvbiAwLjcbWzBtXSBTdXBwb3J0IGZvciBzZXR0
|
193
|
-
aW5nIHVzZXIgYWdlbnQgaGVhZGVyIGFzIHBlciBHaXRodWIgQVBJIHJlcXVp
|
194
|
-
cmVtZW50cwpbG1szMm1WZXJzaW9uIDAuNxtbMG1dIFN1cHBvcnQgZm9yIG1h
|
195
|
-
cmtpbmcgcHJvamVjdHMgYXMgZGVsZXRlZCAocnVuIGZpeGVzL3VwZGF0ZV9k
|
196
|
-
ZWxldGVkLnJiKQpbG1szMm1WZXJzaW9uIDAuNxtbMG1dIE5ldyB0b29sIChn
|
197
|
-
aHQtcmV0cmlldmUtdXNlcikgdG8gcmV0cmlldmUgYWxsIGRhdGEgZm9yIGEg
|
198
|
-
c2luZ2xlIHVzZXIgClsbWzMybVZlcnNpb24gMC43G1swbV0gU3VwcG9ydCBm
|
199
|
-
b3IgcnVubmluZyB3aXRob3V0IGEgY29uZmlnLnlhbWwgZmlsZQpbG1szMm1W
|
200
|
-
ZXJzaW9uIDAuNxtbMG1dIFRlY2huaWNhbCByZXBvcnQgaW4gZG9jLwo=
|
183
|
+
WxtbMzJtVmVyc2lvbiAwLjcuMhtbMG1dIFN1cHBvcnQgZm9yIHJ1bm5pbmcg
|
184
|
+
aW4gc3RhbmRhbG9uZSBtb2RlCg==
|
201
185
|
rdoc_options:
|
202
186
|
- --charset=UTF-8
|
203
187
|
require_paths:
|