greenmonster 0.1.0.pre → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/Readme.markdown +60 -0
- data/lib/greenmonster/spider.rb +87 -52
- data/lib/greenmonster.rb +27 -0
- data/test/test_greenmonster.rb +5 -68
- data/test/test_greenmonster_spider.rb +89 -0
- metadata +12 -33
- data/README.markdown +0 -57
data/Readme.markdown
ADDED
@@ -0,0 +1,60 @@
|
|
1
|
+
Greenmonster
|
2
|
+
============
|
3
|
+
|
4
|
+
Greenmonster is a toolkit for baseball stat enthusiasts or sabermetricians to build a database of play-by-play stats from MLB's [Gameday XML data](http://gd.mlb.com/components/game/).
|
5
|
+
|
6
|
+
The gem currently spiders games from MLB's servers. Over the season, I will be extracting pieces out of my toolkit to parse Gameday XML data to produce play-by-play database of MLB and MiLB stats.
|
7
|
+
|
8
|
+
Usage
|
9
|
+
-----
|
10
|
+
|
11
|
+
If you don't want to specify a download location every time you run the spider, you can set a default games folder location using Greenmonster.set_games_location:
|
12
|
+
|
13
|
+
# Set games folder location
|
14
|
+
>> Greenmonster.set_games_folder('/Users/geoff/games/')
|
15
|
+
|
16
|
+
The spider utility has three public class methods: Spider.pull_game, Spider.pull_day and Spider.pull_days.
|
17
|
+
|
18
|
+
Spider.pull_game takes a game_id (the folder name of the game on the Gameday server) and a hash of options as arguments. If for some reason the game does not fall in the expected folder for the game's date or sport code, you can add those options to the arguments hash. Other options include :games_folder and :print_games (if false, game IDs are not printed to screen).
|
19
|
+
|
20
|
+
# Pulls MLB's 7/4/2011 Toronto @ Boston game
|
21
|
+
>> Greenmonster::Spider.pull_game('gid_2011_07_04_tormlb_bosmlb_1', {:print_games => false})
|
22
|
+
|
23
|
+
Spider.pull_day takes an hash of options as an argument. Greenmonster will create subfolders by MLB "sport_code" (MLB games fall under 'mlb', various minor league games and non-MLB/MiLB games fall under other sport code designations), and then children folders for years, months, days, and specific games.
|
24
|
+
|
25
|
+
# Pulls all MLB games for today
|
26
|
+
>> Greenmonster::Spider.pull_day({:date => Date.today, :games_folder => './home/geoff/games'})
|
27
|
+
|
28
|
+
# Pulls all rookie league games for today
|
29
|
+
>> Greenmonster::Spider.pull_day({:sport_code => 'rok', :date => Date.today, :games_folder => './home/geoff/games'})
|
30
|
+
|
31
|
+
|
32
|
+
Spider.pull_days takes a range of dates to process as an argument, plus a hash of arguments to pass to Spider.pull.
|
33
|
+
|
34
|
+
# Pulls all MLB games for in April, 2012
|
35
|
+
>> Greenmonster::Spider.pull_days((Date.new(2012,4,1)..Date.new(2012,4,30)), {:games_folder => './home/geoff/games'})
|
36
|
+
|
37
|
+
|
38
|
+
Requirements
|
39
|
+
------------
|
40
|
+
- Ruby 1.9
|
41
|
+
- Nokogiri
|
42
|
+
- HTTParty
|
43
|
+
|
44
|
+
Testing
|
45
|
+
-------
|
46
|
+
|
47
|
+
The test suite downloads a few days of data, so it is not fast to execute.
|
48
|
+
|
49
|
+
|
50
|
+
License
|
51
|
+
-------
|
52
|
+
(The MIT License)
|
53
|
+
|
54
|
+
Copyright © [Geoff Harcourt](http://github.com/geoffharcourt) 2012
|
55
|
+
|
56
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ‘Software’), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
57
|
+
|
58
|
+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
59
|
+
|
60
|
+
THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/lib/greenmonster/spider.rb
CHANGED
@@ -1,77 +1,96 @@
|
|
1
1
|
# The Gameday XML Spider utility
|
2
2
|
class Greenmonster::Spider
|
3
3
|
include HTTParty
|
4
|
+
|
5
|
+
##
|
6
|
+
# Pull Gameday XML files for a given game, specified by the game
|
7
|
+
# ID. If date and sport code are not specified as options, these
|
8
|
+
# values are guessed from the game ID string using the home team's
|
9
|
+
# sport code and the date from the scheduled date values in the game
|
10
|
+
# ID.
|
11
|
+
#
|
12
|
+
# Example:
|
13
|
+
# >> Gameday::Spider.pull_game('',{:games_folder => })
|
14
|
+
|
15
|
+
def self.pull_game(game_id,args = {})
|
16
|
+
args = {
|
17
|
+
:date => args[:date] || Date.new(game_id[4,4].to_i, game_id[9,2].to_i, game_id[12,2].to_i),
|
18
|
+
:sport_code => args[:sport_code] || game_id[25,3],
|
19
|
+
:print_games => true,
|
20
|
+
:games_folder => Greenmonster.games_folder
|
21
|
+
}.merge(args)
|
22
|
+
|
23
|
+
puts game_id if args[:print_games]
|
24
|
+
|
25
|
+
raise "Games folder location required." if args[:games_folder].nil?
|
26
|
+
|
27
|
+
paths = {
|
28
|
+
:localGameFolder => "#{args[:games_folder]}/#{args[:sport_code]}/#{format_date_as_folder(args[:date])}/#{game_id}/",
|
29
|
+
:mlbGameFolder => "#{gameday_league_and_date_url(args)}/#{game_id}/"
|
30
|
+
}
|
31
|
+
|
32
|
+
FileUtils.mkdir_p paths[:localGameFolder] + 'inning'
|
33
|
+
|
34
|
+
begin
|
35
|
+
# Always copy linescore first. If we can't get this
|
36
|
+
# data, all other game data is useless.
|
37
|
+
copy_gameday_xml('linescore.xml',paths)
|
38
|
+
|
39
|
+
if args[:date].year > 2007
|
40
|
+
copy_gameday_xml('inning_all.xml',paths)
|
41
|
+
copy_gameday_xml('inning_hit.xml',paths)
|
42
|
+
else
|
43
|
+
# Iterate through the inning files, but skip inning
|
44
|
+
# files numbered 0 (some bad spring training data).
|
45
|
+
# Necessary for games prior to 2008 because there is
|
46
|
+
# no inning_all.xml file in older games.
|
47
|
+
(Nokogiri::XML(self.get("#{paths[:mlbGameFolder]}/inning/").body).search('a')).each do |ic|
|
48
|
+
copy_gameday_xml(ic.attribute('href').value,paths) if ic.attribute('href').value[-3,3] == "xml" unless ic.attribute('href').value[-6,6] == "_0.xml" or ic.attribute('href').value.include?('Score')
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
# Copy base data files
|
53
|
+
# (if inning data wasn't there, this gets skipped)
|
54
|
+
['boxscore.xml','eventLog.xml','players.xml'].each do |file|
|
55
|
+
copy_gameday_xml(file,paths)
|
56
|
+
end
|
57
|
+
rescue StandardError => bang
|
58
|
+
puts "Unable to download some data for #{e.attribute('href').value}"
|
59
|
+
end
|
60
|
+
|
61
|
+
return game_id
|
62
|
+
end
|
4
63
|
|
5
64
|
##
|
6
65
|
# Pull Gameday XML files for a given date. Default options for
|
7
66
|
# the spider are to pull games with sport_code of 'mlb' (games
|
8
67
|
# played by MLB games rather than MiLB teams or foreign teams)
|
9
|
-
# and to pull games on the current date.
|
10
|
-
# :games_folder argument that is the root folder under which
|
11
|
-
# all games will be placed.
|
68
|
+
# and to pull games on the current date.
|
12
69
|
#
|
13
70
|
# Example:
|
14
71
|
# # Pull games from July 4, 2011
|
15
|
-
# >> Gameday::Spider.
|
72
|
+
# >> Gameday::Spider.pull_day({:date => Date.new(2011,7,1), :games_folder => '/Users/geoff/games'})
|
16
73
|
#
|
17
74
|
# Arguments:
|
18
75
|
# args: (Hash)
|
76
|
+
#
|
19
77
|
|
20
78
|
def self.pull_day(args = {})
|
21
79
|
args = {
|
22
80
|
:date => Date.today,
|
23
|
-
:
|
24
|
-
:print_games => true
|
81
|
+
:sport_code => 'mlb',
|
25
82
|
}.merge(args)
|
26
83
|
|
27
|
-
raise "Games folder location required." if args[:games_folder].nil?
|
28
|
-
|
29
|
-
game_day_url_for_date = "http://gd2.mlb.com/components/game/#{args[:league]}/#{format_date_as_folder(args[:date])}"
|
30
|
-
|
31
84
|
# Iterate through every hyperlink on the page.
|
32
85
|
# These links represent the individual game folders
|
33
86
|
# for each date. Reject any links that aren't to game
|
34
87
|
# folders or that are to what look like backup game
|
35
88
|
# folders.
|
36
|
-
(Nokogiri::XML(self.get(
|
37
|
-
|
38
|
-
|
39
|
-
paths = {
|
40
|
-
:localGameFolder => "#{args[:games_folder]}/#{args[:league]}/#{format_date_as_folder(args[:date])}/#{e.attribute('href').value}",
|
41
|
-
:mlbGameFolder => "#{game_day_url_for_date}/#{e.attribute('href').value}"
|
42
|
-
}
|
43
|
-
|
44
|
-
FileUtils.mkdir_p paths[:localGameFolder] + 'inning'
|
45
|
-
|
46
|
-
begin
|
47
|
-
# Always copy linescore first. If we can't get this
|
48
|
-
# data, all other game data is useless.
|
49
|
-
copy_gameday_xml('linescore.xml',paths)
|
50
|
-
|
51
|
-
if args[:date].year > 2007
|
52
|
-
copy_gameday_xml('inning_all.xml',paths)
|
53
|
-
copy_gameday_xml('inning_hit.xml',paths)
|
54
|
-
else
|
55
|
-
# Iterate through the inning files, but skip inning
|
56
|
-
# files numbered 0 (some bad spring training data).
|
57
|
-
# Necessary for games prior to 2008 because there is
|
58
|
-
# no inning_all.xml file in older games.
|
59
|
-
(Nokogiri::XML(self.get("#{paths[:mlbGameFolder]}/inning/").body).search('a')).each do |ic|
|
60
|
-
copy_gameday_xml(ic.attribute('href').value,paths) if ic.attribute('href').value[-3,3] == "xml" unless ic.attribute('href').value[-6,6] == "_0.xml" or ic.attribute('href').value.include?('Score')
|
61
|
-
end
|
62
|
-
end
|
63
|
-
|
64
|
-
# Copy base data files
|
65
|
-
# (if inning data wasn't there, this gets skipped)
|
66
|
-
['boxscore.xml','eventLog.xml','players.xml'].each do |file|
|
67
|
-
copy_gameday_xml(file,paths)
|
68
|
-
end
|
69
|
-
rescue StandardError => bang
|
70
|
-
puts "Unable to download some data for #{e.attribute('href').value}"
|
71
|
-
end
|
89
|
+
(Nokogiri::XML(self.get(gameday_league_and_date_url(args)))/"a").reject{|l| l.attribute('href').value[0,4] != "gid_" or l.attribute('href').value[-5,4] == "_bak"}.each do |e|
|
90
|
+
self.pull_game(e.attribute('href').value.gsub('/',''),args)
|
72
91
|
end
|
73
|
-
|
74
|
-
|
92
|
+
|
93
|
+
return gameday_league_and_date_url(args)
|
75
94
|
end
|
76
95
|
|
77
96
|
##
|
@@ -80,7 +99,7 @@ class Greenmonster::Spider
|
|
80
99
|
#
|
81
100
|
# Example:
|
82
101
|
# # Pull all games in MLB in July 2011
|
83
|
-
# >> Gameday::Spider.pull_days(Date.new(2011,7,1)..Date.new(2011,7,31), {:
|
102
|
+
# >> Gameday::Spider.pull_days(Date.new(2011,7,1)..Date.new(2011,7,31), {:games_folder => '/Users/geoff/games'})
|
84
103
|
#
|
85
104
|
# Arguments:
|
86
105
|
# range: (Range)
|
@@ -91,11 +110,27 @@ class Greenmonster::Spider
|
|
91
110
|
end
|
92
111
|
|
93
112
|
private
|
113
|
+
##
|
114
|
+
# Return the Gameday URL for a given sport_code
|
115
|
+
# and date. Argument must include :date, but
|
116
|
+
# :sport_code will be assumed to be 'mlb' if not
|
117
|
+
# specified.
|
118
|
+
#
|
119
|
+
# Arguments:
|
120
|
+
# args: (Hash)
|
121
|
+
#
|
122
|
+
def self.gameday_league_and_date_url(args = {})
|
123
|
+
raise "Date required." unless args[:date]
|
124
|
+
args[:sport_code] ||= 'mlb'
|
125
|
+
|
126
|
+
"http://gd2.mlb.com/components/game/#{args[:sport_code]}/#{format_date_as_folder(args[:date])}"
|
127
|
+
end
|
128
|
+
|
94
129
|
##
|
95
130
|
# Copy XML files from the Gameday severs to your
|
96
131
|
# local machine for parsing and analysis. This method
|
97
|
-
# is used by Spider
|
98
|
-
# machine in a similar layout to the one used by
|
132
|
+
# is used by Spider pull class methods to put files on
|
133
|
+
# your local machine in a similar layout to the one used by
|
99
134
|
# Gameday. The paths argument gets built by Spider.pull
|
100
135
|
# during the pull process.
|
101
136
|
#
|
@@ -113,7 +148,7 @@ class Greenmonster::Spider
|
|
113
148
|
# Output a folder format similar to the one used by Gameday.
|
114
149
|
#
|
115
150
|
# Example:
|
116
|
-
# Spider.format_date_as_folder(Date.new(2011,7,4))
|
151
|
+
# >> Spider.format_date_as_folder(Date.new(2011,7,4))
|
117
152
|
# => "year_2011/month_07/day_04"
|
118
153
|
#
|
119
154
|
# Arguments:
|
data/lib/greenmonster.rb
CHANGED
@@ -3,8 +3,35 @@ require 'nokogiri'
|
|
3
3
|
require 'fileutils'
|
4
4
|
|
5
5
|
module Greenmonster
|
6
|
+
@@games_folder = nil
|
6
7
|
|
8
|
+
##
|
9
|
+
# Set the default folder to which games
|
10
|
+
# are saved after being downloaded from
|
11
|
+
# the server.
|
12
|
+
#
|
13
|
+
# Example:
|
14
|
+
# => Greenmonster.set_games_folder("/Users/geoff/game_data")
|
15
|
+
#
|
16
|
+
# Arguments:
|
17
|
+
# location: (String)
|
18
|
+
#
|
7
19
|
|
20
|
+
def self.set_games_folder(location)
|
21
|
+
@@games_folder = location
|
22
|
+
end
|
23
|
+
|
24
|
+
##
|
25
|
+
# Return the default games folder location
|
26
|
+
#
|
27
|
+
# Example:
|
28
|
+
# >> Greenmonster.set_games_folder("/Users/geoff/game_data")
|
29
|
+
# >> Greenmonster.games_folder
|
30
|
+
# => "/Users/geoff/game_data"
|
31
|
+
|
32
|
+
def self.games_folder
|
33
|
+
@@games_folder
|
34
|
+
end
|
8
35
|
end
|
9
36
|
|
10
37
|
require 'greenmonster/spider'
|
data/test/test_greenmonster.rb
CHANGED
@@ -1,78 +1,15 @@
|
|
1
1
|
require 'minitest/autorun'
|
2
2
|
require 'greenmonster'
|
3
3
|
|
4
|
-
class
|
4
|
+
class GreenmonsterTest < MiniTest::Unit::TestCase
|
5
5
|
def setup
|
6
|
-
@local_test_data_location = "./greenmonster_test_games"
|
7
|
-
FileUtils.mkdir_p @local_test_data_location
|
8
6
|
end
|
9
7
|
|
10
|
-
def
|
11
|
-
|
12
|
-
assert_equal
|
13
|
-
end
|
14
|
-
|
15
|
-
def test_copy_gameday_xml
|
16
|
-
paths = {
|
17
|
-
:localGameFolder => "#{@local_test_data_location}/mlb/year_2011/month_05/day_03/gid_2011_05_03_sfnmlb_nynmlb_1/",
|
18
|
-
:mlbGameFolder => "http://gd2.mlb.com/components/game/mlb/year_2011/month_05/day_03/gid_2011_05_03_sfnmlb_nynmlb_1/"
|
19
|
-
}
|
20
|
-
FileUtils.mkdir_p paths[:localGameFolder] + 'inning'
|
21
|
-
|
22
|
-
%w(linescore.xml inning_all.xml).each do |f|
|
23
|
-
Greenmonster::Spider.copy_gameday_xml(f,paths)
|
24
|
-
end
|
25
|
-
|
26
|
-
assert_equal 287337, Nokogiri::XML(open(paths[:localGameFolder] + 'linescore.xml')).search("game").first.attribute('game_pk').value.to_i
|
27
|
-
assert_equal 10, Nokogiri::XML(open(paths[:localGameFolder] + 'inning/inning_all.xml')).search("inning").count
|
28
|
-
assert_equal 400023, Nokogiri::XML(open(paths[:localGameFolder] + 'inning/inning_all.xml')).search("atbat").first.attribute('batter').value.to_i
|
29
|
-
end
|
30
|
-
|
31
|
-
def test_pull_day
|
32
|
-
Greenmonster::Spider.pull_day({:print_games => false, :games_folder => @local_test_data_location, :date => Date.new(2011,6,7)})
|
33
|
-
|
34
|
-
assert_equal 17, Dir.entries(@local_test_data_location + '/mlb/year_2011/month_06/day_07/').count
|
35
|
-
|
36
|
-
game_location = @local_test_data_location + '/mlb/year_2011/month_06/day_07/gid_2011_06_07_bosmlb_nyamlb_1'
|
37
|
-
%w(linescore.xml boxscore.xml players.xml eventLog.xml inning).each do |f|
|
38
|
-
assert Dir.entries(game_location).include? f
|
39
|
-
end
|
40
|
-
|
41
|
-
boxscore = Nokogiri::XML(open(game_location + '/boxscore.xml'))
|
42
|
-
eventlog = Nokogiri::XML(open(game_location + '/eventLog.xml'))
|
43
|
-
players = Nokogiri::XML(open(game_location + '/players.xml'))
|
44
|
-
innings = Nokogiri::XML(open(game_location + '/inning/inning_all.xml'))
|
45
|
-
|
46
|
-
assert_equal 8, boxscore.search('pitcher').count
|
47
|
-
assert_equal 147, boxscore.search('boxscore').first.attribute('home_id').value.to_i
|
48
|
-
assert eventlog.search("event[number='19']").attribute('description').value.include?('Pedroia walks.')
|
49
|
-
assert_equal 'Magadan', players.search("team[type='away']").search("coach[position='batting_coach']").attribute('last').value
|
50
|
-
assert_equal 'Strikeout', innings.search('atbat').last.attribute('event').value
|
51
|
-
end
|
52
|
-
|
53
|
-
def test_pull_minor_league_games
|
54
|
-
Greenmonster::Spider.pull_day({:league => 'aaa', :print_games => false, :games_folder => @local_test_data_location, :date => Date.new(2011,9,1)})
|
55
|
-
|
56
|
-
assert_equal 13, Dir.entries(@local_test_data_location + '/aaa/year_2011/month_09/day_01/').count
|
57
|
-
%w(linescore.xml boxscore.xml players.xml eventLog.xml inning).each do |f|
|
58
|
-
assert Dir.entries(@local_test_data_location + '/aaa/year_2011/month_09/day_01/gid_2011_09_01_albaaa_mrbaaa_1/').include? f
|
59
|
-
end
|
60
|
-
end
|
61
|
-
|
62
|
-
def test_pull_games_prior_to_2008
|
63
|
-
Greenmonster::Spider.pull_day({:print_games => false, :games_folder => @local_test_data_location, :date => Date.new(2007,4,15)})
|
64
|
-
|
65
|
-
assert_equal 12, Dir.entries(@local_test_data_location + '/mlb/year_2007/month_04/day_15/gid_2007_04_15_detmlb_tormlb_1/inning/').count
|
66
|
-
end
|
67
|
-
|
68
|
-
def test_pull_days
|
69
|
-
Greenmonster::Spider.pull_days((Date.new(2011,8,4)..Date.new(2011,8,5)), {:print_games => false, :games_folder => @local_test_data_location})
|
70
|
-
|
71
|
-
assert_equal 4, Dir.entries(@local_test_data_location + '/mlb/year_2011/month_08/').count
|
72
|
-
assert_equal 17, Dir.entries(@local_test_data_location + '/mlb/year_2011/month_08/day_05').count
|
8
|
+
def test_set_games_folder_location
|
9
|
+
Greenmonster.set_games_folder('./test_greenmonster_folder')
|
10
|
+
assert_equal './test_greenmonster_folder', Greenmonster.games_folder
|
73
11
|
end
|
74
12
|
|
75
13
|
def teardown
|
76
|
-
FileUtils.remove_dir @local_test_data_location
|
77
14
|
end
|
78
|
-
end
|
15
|
+
end
|
@@ -0,0 +1,89 @@
|
|
1
|
+
require 'minitest/autorun'
|
2
|
+
require 'greenmonster'
|
3
|
+
|
4
|
+
class GreenmonsterSpiderTest < MiniTest::Unit::TestCase
|
5
|
+
def setup
|
6
|
+
@local_test_data_location = "./greenmonster_test_games"
|
7
|
+
FileUtils.mkdir_p @local_test_data_location
|
8
|
+
end
|
9
|
+
|
10
|
+
def test_format_date_as_folder
|
11
|
+
assert_equal "year_2011/month_07/day_04", Greenmonster::Spider.format_date_as_folder(Date.new(2011,7,4))
|
12
|
+
assert_equal "year_2012/month_12/day_31", Greenmonster::Spider.format_date_as_folder(Date.new(2012,12,31))
|
13
|
+
end
|
14
|
+
|
15
|
+
def test_copy_gameday_xml
|
16
|
+
paths = {
|
17
|
+
:localGameFolder => "#{@local_test_data_location}/mlb/year_2011/month_05/day_03/gid_2011_05_03_sfnmlb_nynmlb_1/",
|
18
|
+
:mlbGameFolder => "http://gd2.mlb.com/components/game/mlb/year_2011/month_05/day_03/gid_2011_05_03_sfnmlb_nynmlb_1/"
|
19
|
+
}
|
20
|
+
FileUtils.mkdir_p paths[:localGameFolder] + 'inning'
|
21
|
+
|
22
|
+
%w(linescore.xml inning_all.xml).each do |f|
|
23
|
+
Greenmonster::Spider.copy_gameday_xml(f,paths)
|
24
|
+
end
|
25
|
+
|
26
|
+
assert_equal 287337, Nokogiri::XML(open(paths[:localGameFolder] + 'linescore.xml')).search("game").first.attribute('game_pk').value.to_i
|
27
|
+
assert_equal 10, Nokogiri::XML(open(paths[:localGameFolder] + 'inning/inning_all.xml')).search("inning").count
|
28
|
+
assert_equal 400023, Nokogiri::XML(open(paths[:localGameFolder] + 'inning/inning_all.xml')).search("atbat").first.attribute('batter').value.to_i
|
29
|
+
end
|
30
|
+
|
31
|
+
def test_pull_day
|
32
|
+
Greenmonster::Spider.pull_day({:print_games => false, :games_folder => @local_test_data_location, :date => Date.new(2011,6,7)})
|
33
|
+
|
34
|
+
assert_equal 17, Dir.entries(@local_test_data_location + '/mlb/year_2011/month_06/day_07/').count
|
35
|
+
|
36
|
+
game_location = @local_test_data_location + '/mlb/year_2011/month_06/day_07/gid_2011_06_07_bosmlb_nyamlb_1'
|
37
|
+
%w(linescore.xml boxscore.xml players.xml eventLog.xml inning).each do |f|
|
38
|
+
assert Dir.entries(game_location).include? f
|
39
|
+
end
|
40
|
+
|
41
|
+
boxscore = Nokogiri::XML(open(game_location + '/boxscore.xml'))
|
42
|
+
eventlog = Nokogiri::XML(open(game_location + '/eventLog.xml'))
|
43
|
+
players = Nokogiri::XML(open(game_location + '/players.xml'))
|
44
|
+
innings = Nokogiri::XML(open(game_location + '/inning/inning_all.xml'))
|
45
|
+
|
46
|
+
assert_equal 8, boxscore.search('pitcher').count
|
47
|
+
assert_equal 147, boxscore.search('boxscore').first.attribute('home_id').value.to_i
|
48
|
+
assert eventlog.search("event[number='19']").attribute('description').value.include?('Pedroia walks.')
|
49
|
+
assert_equal 'Magadan', players.search("team[type='away']").search("coach[position='batting_coach']").attribute('last').value
|
50
|
+
assert_equal 'Strikeout', innings.search('atbat').last.attribute('event').value
|
51
|
+
end
|
52
|
+
|
53
|
+
def test_pull_non_mlb_sport_code_games
|
54
|
+
Greenmonster::Spider.pull_day({:sport_code => 'aaa', :print_games => false, :games_folder => @local_test_data_location, :date => Date.new(2011,9,1)})
|
55
|
+
|
56
|
+
assert_equal 13, Dir.entries(@local_test_data_location + '/aaa/year_2011/month_09/day_01/').count
|
57
|
+
%w(linescore.xml boxscore.xml players.xml eventLog.xml inning).each do |f|
|
58
|
+
assert Dir.entries(@local_test_data_location + '/aaa/year_2011/month_09/day_01/gid_2011_09_01_albaaa_mrbaaa_1/').include? f
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
def test_pull_games_prior_to_2008
|
63
|
+
Greenmonster::Spider.pull_day({:print_games => false, :games_folder => @local_test_data_location, :date => Date.new(2007,4,15)})
|
64
|
+
|
65
|
+
assert_equal 12, Dir.entries(@local_test_data_location + '/mlb/year_2007/month_04/day_15/gid_2007_04_15_detmlb_tormlb_1/inning/').count
|
66
|
+
end
|
67
|
+
|
68
|
+
def test_pull_days
|
69
|
+
Greenmonster::Spider.pull_days((Date.new(2011,8,4)..Date.new(2011,8,5)), {:print_games => false, :games_folder => @local_test_data_location})
|
70
|
+
|
71
|
+
assert_equal 4, Dir.entries(@local_test_data_location + '/mlb/year_2011/month_08/').count
|
72
|
+
assert_equal 17, Dir.entries(@local_test_data_location + '/mlb/year_2011/month_08/day_05').count
|
73
|
+
end
|
74
|
+
|
75
|
+
def test_pull_day_with_default_folder_location
|
76
|
+
Greenmonster.set_games_folder(@local_test_data_location)
|
77
|
+
Greenmonster::Spider.pull_day({:date => Date.new(2011,7,4), :print_games => false})
|
78
|
+
|
79
|
+
assert_equal 288186, Nokogiri::XML(open(@local_test_data_location + '/mlb/year_2011/month_07/day_04/gid_2011_07_04_tormlb_bosmlb_1/boxscore.xml')).search('boxscore').first.attribute('game_pk').value.to_i
|
80
|
+
end
|
81
|
+
|
82
|
+
def test_pull_single_game_by_game_id
|
83
|
+
Greenmonster::Spider.pull_game('gid_2011_07_04_tormlb_bos_mlb_1', {:games_folder => @local_test_data_location, :print_games => false})
|
84
|
+
end
|
85
|
+
|
86
|
+
def teardown
|
87
|
+
FileUtils.remove_dir @local_test_data_location
|
88
|
+
end
|
89
|
+
end
|
metadata
CHANGED
@@ -1,50 +1,29 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: greenmonster
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.0
|
5
|
-
prerelease:
|
4
|
+
version: 0.1.0
|
5
|
+
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
8
8
|
- Geoff Harcourt
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-03-
|
13
|
-
dependencies:
|
14
|
-
|
15
|
-
name: nokogiri
|
16
|
-
requirement: &70119326330320 !ruby/object:Gem::Requirement
|
17
|
-
none: false
|
18
|
-
requirements:
|
19
|
-
- - ! '>='
|
20
|
-
- !ruby/object:Gem::Version
|
21
|
-
version: 1.5.2
|
22
|
-
type: :runtime
|
23
|
-
prerelease: false
|
24
|
-
version_requirements: *70119326330320
|
25
|
-
- !ruby/object:Gem::Dependency
|
26
|
-
name: httparty
|
27
|
-
requirement: &70119326329200 !ruby/object:Gem::Requirement
|
28
|
-
none: false
|
29
|
-
requirements:
|
30
|
-
- - ! '>='
|
31
|
-
- !ruby/object:Gem::Version
|
32
|
-
version: 0.8.1
|
33
|
-
type: :runtime
|
34
|
-
prerelease: false
|
35
|
-
version_requirements: *70119326329200
|
36
|
-
description: A utility for spidering and parsing MLB Gameday XML files.
|
12
|
+
date: 2012-03-13 00:00:00.000000000 Z
|
13
|
+
dependencies: []
|
14
|
+
description: A utility for working with MLB Gameday XML data.
|
37
15
|
email: geoff.harcourt@gmail.com
|
38
16
|
executables: []
|
39
17
|
extensions: []
|
40
18
|
extra_rdoc_files: []
|
41
19
|
files:
|
42
20
|
- Rakefile
|
43
|
-
-
|
44
|
-
- lib/greenmonster/spider.rb
|
21
|
+
- Readme.markdown
|
45
22
|
- lib/greenmonster.rb
|
23
|
+
- lib/greenmonster/spider.rb
|
46
24
|
- test/test_greenmonster.rb
|
47
|
-
|
25
|
+
- test/test_greenmonster_spider.rb
|
26
|
+
homepage: http://github.com/geoffharcourt/greenmonster
|
48
27
|
licenses: []
|
49
28
|
post_install_message:
|
50
29
|
rdoc_options: []
|
@@ -59,13 +38,13 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
59
38
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
60
39
|
none: false
|
61
40
|
requirements:
|
62
|
-
- - ! '
|
41
|
+
- - ! '>='
|
63
42
|
- !ruby/object:Gem::Version
|
64
|
-
version:
|
43
|
+
version: '0'
|
65
44
|
requirements: []
|
66
45
|
rubyforge_project:
|
67
46
|
rubygems_version: 1.8.11
|
68
47
|
signing_key:
|
69
48
|
specification_version: 3
|
70
|
-
summary: A utility for
|
49
|
+
summary: A utility for working with MLB Gameday XML data.
|
71
50
|
test_files: []
|
data/README.markdown
DELETED
@@ -1,57 +0,0 @@
|
|
1
|
-
Greenmonster
|
2
|
-
============
|
3
|
-
|
4
|
-
Greenmonster is a toolkit for baseball stat enthusiasts or sabermetricians to build a database of
|
5
|
-
play-by-play stats from MLB's [Gameday XML data](http://gd.mlb.com/components/game/).
|
6
|
-
|
7
|
-
The pre-release gem is currently spiders games from MLB's servers. I will be extracting pieces out
|
8
|
-
of my toolkit to parse Gameday XML data to produce play-by-play database of MLB and MiLB stats.
|
9
|
-
|
10
|
-
Usage
|
11
|
-
-----
|
12
|
-
|
13
|
-
The spider utility has two public class methods: Spider.pull_day and Spider.pull_days.
|
14
|
-
|
15
|
-
Spider.pull_day takes an hash of options as an argument. The only mandatory argument option
|
16
|
-
is to specify :games_folder, which is the local folder where you want to store the game files.
|
17
|
-
Greenmonster will create subfolders by MLB "sport_code" (MLB games fall under 'mlb', various
|
18
|
-
minor league games and non-MLB/MiLB games fall under other sport code designations), and then
|
19
|
-
children folders for years, months, days, and specific games.
|
20
|
-
|
21
|
-
# Pulls all MLB games for today
|
22
|
-
=> Greenmonster::Spider.pull_day({:date => Date.today, :games_folder => './home/geoff/games'})
|
23
|
-
|
24
|
-
# Pulls all rookie league games for today
|
25
|
-
> Greenmonster::Spider.pull_day({:sport_code => 'rok', :date => Date.today, :games_folder => './home/geoff/games'})
|
26
|
-
|
27
|
-
|
28
|
-
Spider.pull_days takes a range of dates to process as an argument, plus a hash of arguments to pass
|
29
|
-
to Spider.pull.
|
30
|
-
|
31
|
-
# Pulls all MLB games for in April, 2012
|
32
|
-
=> Greenmonster::Spider.pull_days((Date.new(2012,4,1)..Date.new(2012,4,30)), {:games_folder => './home/geoff/games'})
|
33
|
-
|
34
|
-
|
35
|
-
Requirements
|
36
|
-
------------
|
37
|
-
- Ruby 1.9
|
38
|
-
- Nokogiri
|
39
|
-
- HTTParty
|
40
|
-
|
41
|
-
Testing
|
42
|
-
-------
|
43
|
-
|
44
|
-
The test suite downloads a few days of data, so it is not fast to execute.
|
45
|
-
|
46
|
-
|
47
|
-
License
|
48
|
-
-------
|
49
|
-
(The MIT License)
|
50
|
-
|
51
|
-
Copyright © [Geoff Harcourt](http://github.com/geoffharcourt) 2012
|
52
|
-
|
53
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ‘Software’), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
54
|
-
|
55
|
-
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
56
|
-
|
57
|
-
THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|