greenmonster 0.4.0.dev → 0.5.0

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/README.markdown

@@ -1,14 +1,9 @@
 Greenmonster
 ============
 
-Greenmonster is a toolkit for baseball stat enthusiasts or sabermetricians to build a database of play-by-play stats from MLB's [Gameday XML data](http://gd.mlb.com/components/game/).
+Greenmonster is a toolkit for baseball stat enthusiasts or sabermetricians to build a database of play-by-play stats from MLB's [Gameday XML data](http://gd.mlb.com/components/game/). The current tool provides the ability to spider Gameday XML data from MLB's servers for personal research. Future iterations of the tool will provide the ability to parse the data and store it in a SQL database.
 
-The provides three tools:
-* Spidering of MLB/MiLB games
-* Parsing of Gameday XML
-* Mixin methods you can use to extend your own classes
-
-Usage 
+Usage
 =====
 
 Spider
@@ -21,66 +16,39 @@ If you don't want to specify a download location every time you run the spider,
 Greenmonster.set_games_folder('/Users/geoff/games/')
 ```
 
-The spider utility has three public class methods: Spider.pull_game, Spider.pull_day, and Spider.pull_days. 
+The spider utility has three public class methods: Spider.pull_game, Spider.pull_day, and Spider.pull_days.
 
-Spider.pull_game takes a game_id (the folder name of the game on the Gameday server) and a hash of options as arguments. If for some reason the game does not fall in the expected folder for the game's date or sport code, you can add those options to the arguments hash. Other options include :games_folder and :print_games (if false, game IDs are not printed to screen).
+Spider.pull_game takes a game_id (the folder name of the game on the Gameday server) and the date. The date is necessary because if a game is postponed or (yes, it's happened this decade) preponed, the game ID might have a date different than the actual date on which the game was played.
 
 ```ruby
-# Pulls MLB's 7/4/2011 Toronto @ Boston game
-Greenmonster::Spider.pull_game('gid_2011_07_04_tormlb_bosmlb_1', {:print_games => false})
+# Pull MLB's 7/4/2011 Toronto @ Boston game
+Greenmonster::Spider.pull_game('gid_2011_07_04_tormlb_bosmlb_1', Date.new(2011,7,4))
 ```
 
 Spider.pull_day takes an hash of options as an argument. Greenmonster will create subfolders by MLB "sport_code" (MLB games fall under 'mlb', various minor league games and non-MLB/MiLB games fall under other sport code designations), and then children folders for years, months, days, and specific games. Sport code can be a string or an array of sport code strings.
 
 ```ruby
-# Pulls all MLB games for today
-Greenmonster::Spider.pull_day({:date => Date.today, :games_folder => './home/geoff/games'})
-
-# Pulls all rookie league games for today
-Greenmonster::Spider.pull_day({:sport_code => 'rok', :date => Date.today, :games_folder => './home/geoff/games'})
+# Pull all MLB games for today
+Greenmonster::Spider.pull_day(Date.today, 'mlb')
 
-# Pulls all games in all sport codes for today
-Greenmonster::Spider.pull_day({:all_sport_codes => true, :date => Date.today, :games_folder => './home/geoff/games'})
+# Pull all rookie league games for today
+Greenmonster::Spider.pull_day(Date.today, 'rok')
 
-# Pulls all games in rookie and winter league games for January 2nd, 2010
-Greenmonster::Spider.pull_day({:sport_code => ['rok','win'], :date => Date.new(2012,1,2), :games_folder => './home/geoff/games'})
+# Pull all games in all sport codes for today
+Greenmonster::SPORT_CODES.each do |sport_code|
+  Greenmonster::Spider.pull_day(Date.today, sport_code)
+end
 ```
 
 
 
-Spider.pull_days takes a range of dates to process as an argument, plus a hash of arguments to pass to Spider.pull.
+Spider.pull_days takes a range of dates to process as an argument, plus the sport code for the games (MLB.
 
 ```ruby
-# Pulls all MLB games for in April, 2012
-Greenmonster::Spider.pull_days((Date.new(2012,4,1)..Date.new(2012,4,30)), {:games_folder => './home/geoff/games'})
+# Pull all MLB games for in April, 2012
+Greenmonster::Spider.pull_days((Date.new(2012,4,1)..Date.new(2012,4,30)), 'mlb')
 ```	
 
-Mixins (ALPHA)
---------------
-(Under development.)
-
-As of version 0.4.0, Greenmonster provides the Greenmonster::Player module which can be used to extend any Ruby class you use that represents players. Include the module in your class to get Greenmonster-specific functionality like parsing players out of games.
-
-```ruby
-class MlbPlayer < ActiveRecord::Base
-   include Greenmonster::Player
-end
-
->> MlbPlayer.create_from_gameday_xml_game('gid_2011_07_04_tormlb_bosmlb_1')
-```
-
-Migrations (ALPHA)
-------------------
-
-WARNING: THIS FEATURE IS UNDER DEVELOPMENT. USE AT YOUR OWN RISK. 
-ANOTHER WARNING: Using migrations on alpha/beta/pre versions of Greenmonster may not be compatible with formal releases that come later.
-
-If you use ActiveRecord, Greenmonster provides a generator that can generate tables for Greenmonster data. Add Greenmonster to your Gemfile:
-```ruby
-gem 'greenmonster', '~> 0.4.0'
-```
-
-After you pull the gem in with Bundler, you will have access to Greenmonster generators. The Install generator attempts to install a set of standard name tables that correspond to Greenmonster data. 
 
 
 Requirements
@@ -89,19 +57,18 @@ Requirements
 - Bundler
 - Nokogiri
 - HTTParty
-- ActiveRecord (if you want to use migration generators or any mixins that involve AR saves)
 
 Testing
 -------
 
-The test suite downloads a few days of data, so it is not fast to execute.
+The test suite is being migrated to RSpec, and uses bourne.
 
 
 License
 -------
 (The MIT License)
 
-Copyright &copy; [Geoff Harcourt](http://github.com/geoffharcourt) 2012
+Copyright &copy; [Geoff Harcourt](http://github.com/geoffharcourt) 2012-2013
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ‘Software’), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

data/Rakefile

@@ -1,9 +1,12 @@
 require "bundler/gem_tasks"
 require 'rake/testtask'
+require 'rspec/core/rake_task'
 
 Rake::TestTask.new do |t|
   t.libs << 'test'
 end
 
+RSpec::Core::RakeTask.new('spec')
+
 desc "Run Tests"
 task :default => :test

data/greenmonster.gemspec

@@ -1,13 +1,13 @@
 # -*- encoding: utf-8 -*-
-$:.push File.expand_path("../lib", __FILE__)
-require "greenmonster/version"
+$:.push File.expand_path('../lib', __FILE__)
+require 'greenmonster/version'
 
 Gem::Specification.new do |s|
-  s.name        = "greenmonster"
+  s.name        = 'greenmonster'
   s.version     = Greenmonster::VERSION
-  s.authors     = ["Geoff Harcourt"]
-  s.email       = ["geoff.harcourt@gmail.com"]
-  s.homepage    = "http://github.com/geoffharcourt/greenmonster"
+  s.authors     = ['Geoff Harcourt']
+  s.email       = ['geoff.harcourt@gmail.com']
+  s.homepage    = 'http://github.com/geoffharcourt/greenmonster'
   s.summary     = %q{A utility for working with MLB Gameday XML data.}
   s.description = %q{A utility for working with MLB Gameday XML data.}
 
@@ -16,10 +16,9 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
 
-  s.add_dependency "nokogiri"
-  s.add_dependency "httparty"
-  s.add_dependency "activerecord"
-  s.add_development_dependency "minitest"
-  s.add_development_dependency "supermodel"
-  s.add_development_dependency "sqlite3-ruby"
-end
+  s.add_dependency 'nokogiri'
+  s.add_dependency 'httparty'
+  s.add_development_dependency 'rspec'
+  s.add_development_dependency 'bourne'
+  s.add_development_dependency 'rake'
+end

data/lib/greenmonster.rb

@@ -3,26 +3,26 @@ require "bundler/setup"
 
 require 'httparty'
 require 'nokogiri'
-require 'pathname'
 require 'fileutils'
-require 'active_record'
 require 'date'
 
 module Greenmonster
+  SPORT_CODES = %w(aaa aax afa afx asx bbc fps hsb ind int jml nae naf nas nat naw oly rok win)
+
   @@games_folder = nil
-  
+
   ##
   # Set the default folder to which games
   # are saved after being downloaded from
   # the server.
   #
-  # Example: 
-  #   => Greenmonster.set_games_folder("/Users/geoff/game_data")
+  # Example:
+  #  => Greenmonster.set_games_folder("/Users/geoff/game_data")
   #
   # Arguments:
-  #   location: (String)
+  #  location: (String)
   #
-  
+
   def self.set_games_folder(location)
     @@games_folder = Pathname.new(location)
   end
@@ -31,57 +31,61 @@ module Greenmonster
   # Return the default games folder location
   #
   # Example:
-  #   >> Greenmonster.set_games_folder("/Users/geoff/game_data")
-  #   >> Greenmonster.games_folder
-  #   => #<Pathname:/Users/geoff/game_data> 
+  #  >> Greenmonster.set_games_folder("/Users/geoff/game_data")
+  #  >> Greenmonster.games_folder
+  #  => #<Pathname:/Users/geoff/game_data>
 
   def self.games_folder
     @@games_folder
   end
-  
-  def self.format_date_as_folder(date)
-    date.strftime("year_%Y/month_%m/day_%d")
-  end
-  
+
   ##
   # Walk the dates in a range of dates, and execute whatever
-  # methods on the date and argument set specified. Used when 
+  # methods on the date and argument set specified. Used when
   # processing games and players.
   #
   #
-  
-  def self.traverse_dates(range = (Date.today..Date.today), args = {})
-     range.each do |day|
-        yield day,args
-     end
+
+  def self.traverse_dates(range, args)
+    range.each { |day| yield day,args }
   end
-  
+
   ##
   # Walk the game folders in a range of dates, and execute whatever
-  # methods on the game folder and argument set specified. Used when 
+  # methods on the game folder and argument set specified. Used when
   # processing games and players.
   #
   #
-  
-  def self.traverse_folders_for_date(day,args = {})
-     begin
-        folders = Dir.entries(Pathname.new(args[:games_folder] || @@games_folder) + (args[:sport_code] || 'mlb') + self.format_date_as_folder(day))
-     rescue StandardError => boom
-        puts "No files for #{day.to_s}."
-     end
-
-     unless folders.nil?
-        folders.sort.each do |gdir|
-           if gdir[0,3] == 'gid' and gdir[-4,4] != "_bak"
-              yield gdir, args
-           end
-        end
-     end
+
+  def self.traverse_folders_for_date(date, args)
+    game_folders_for_date_and_sport_code(date, args[:sport_code]).each do |gdir|
+      yield gdir, args
+    end
   end
+
+
+  private
+
+  def self.game_folders_for_date_and_sport_code(date, sport_code)
+    folders_for_date_and_sport_code(date, sport_code).select do |folder|
+      folder[0,3] == 'gid' && folder[-4,4] != '_bak'
+    end
+  end
+
+  def self.folders_for_date_and_sport_code(date, sport_code)
+    begin
+      Dir.entries(
+        Pathname.new(games_folder) + sport_code + format_date_as_folder(date)
+      ).sort
+    rescue Errno::ENOENT
+      []
+    end
+  end
+
+  def self.format_date_as_folder(date)
+    date.strftime("year_%Y/month_%m/day_%d")
+  end
+
 end
 
-require 'greenmonster/spider'
-require 'greenmonster/model_extensions/player'
-require 'greenmonster/model_extensions/mlb_game'
-require 'greenmonster/model_extensions/mlb_probable_pitcher'
-require 'greenmonster/parser'
+require 'greenmonster/spider'

data/lib/greenmonster/spider.rb

@@ -1,107 +1,43 @@
 # The Gameday XML Spider utility
-module Greenmonster::Spider
+class Greenmonster::Spider
   include HTTParty
-  
+
   ##
-  # Pull Gameday XML files for a given game, specified by the game 
+  # Pull Gameday XML files for a given game, specified by the game
   # ID. If date and sport code are not specified as options, these
   # values are guessed from the game ID string using the home team's
   # sport code and the date from the scheduled date values in the game
   # ID.
   #
   # Example:
-  #   >> Gameday::Spider.pull_game('',{:games_folder => })
-  
-  def self.pull_game(game_id,args = {})
-    args = {
-      :date => args[:date] || Date.new(game_id[4,4].to_i, game_id[9,2].to_i, game_id[12,2].to_i),
-      :sport_code => args[:sport_code] || game_id[25,3],
-      :print_games => true,
-      :games_folder => Greenmonster.games_folder
-    }.merge(args)
-    raise "Games folder location required." if args[:games_folder].nil?
-    
-    args[:games_folder] = Pathname.new(args[:games_folder])
-    
-    puts game_id if args[:print_games]
-        
-    paths = {
-      :localGameFolder => args[:games_folder] + args[:sport_code] + format_date_as_folder(args[:date]) + game_id,
-      :mlbGameFolder => "#{gameday_league_and_date_url(args)}/#{game_id}/"
-    }
-        
-    FileUtils.mkdir_p paths[:localGameFolder] + 'inning'
-
-    begin
-      # Always copy linescore first. If we can't get this
-      # data, all other game data is useless.
-      copy_gameday_xml('linescore.xml',paths)
-      
-      if args[:date].year > 2007
-        copy_gameday_xml('inning_all.xml',paths)
-        copy_gameday_xml('inning_hit.xml',paths)
-      else
-        # Iterate through the inning files, but skip inning 
-        # files numbered 0 (some bad spring training data).
-        # Necessary for games prior to 2008 because there is
-        # no inning_all.xml file in older games.
-        (Nokogiri::XML(self.get("#{paths[:mlbGameFolder]}/inning/").body).search('a')).each do |ic|
-          copy_gameday_xml(ic.attribute('href').value,paths) if ic.attribute('href').value[-3,3]  == "xml" unless ic.attribute('href').value[-6,6] == "_0.xml" or ic.attribute('href').value.include?('Score')
-        end
-      end
+  #  >> Gameday::Spider.pull_game('',{:games_folder => })
 
-      # Copy base data files 
-      # (if inning data wasn't there, this gets skipped)
-      ['boxscore.xml','eventLog.xml','players.xml'].each do |file|
-        copy_gameday_xml(file,paths)
-      end
-    rescue StandardError => bang
-      puts "Unable to download some data for #{game_id}"  
+  def pull_game(game_id, date)
+    make_folders_for_game(game_id, date)
+
+    %w(boxscore.xml game_events.xml inning_all.xml linescore.xml players.xml).each do |file_name|
+      copy_gameday_xml(game_id, date, file_name)
     end
-    
-    return game_id
   end
-    
+
   ##
-  # Pull Gameday XML files for a given date. Default options for 
-  # the spider are to pull games with sport_code of 'mlb' (games 
-  # played by MLB games rather than MiLB teams or foreign teams) 
-  # and to pull games on the current date. 
+  # Pull Gameday XML files for a given date. Default options for
+  # the spider are to pull games with sport_code of 'mlb' (games
+  # played by MLB games rather than MiLB teams or foreign teams)
+  # and to pull games on the current date.
   #
   # Example:
-  #   # Pull games from July 4, 2011
-  #   >> Gameday::Spider.pull_day({:date => Date.new(2011,7,1), :games_folder => '/Users/geoff/games'})
+  #  # Pull games from July 4, 2011
+  #  >> Gameday::Spider.pull_day({:date => Date.new(2011,7,1), :games_folder => '/Users/geoff/games'})
   #
   # Arguments:
-  #   args: (Hash)
+  #  args: (Hash)
   #
-    
-  def self.pull_day(args = {})
-    args = {
-      :date => Date.today,
-      :sport_code => 'mlb',
-    }.merge(args)
-    
-    # If we want all sport codes, set up the array.
-    if args[:all_sport_codes]
-      args[:sport_codes] = %w(aaa aax afa afx asx bbc fps hsb ind int jml nae naf nas nat naw oly rok win)
-    else
-      args[:sport_codes] = [args[:sport_code] || 'mlb'].flatten
-    end
-    
-    # Iterate through every hyperlink on the page.
-    # These links represent the individual game folders
-    # for each date. Reject any links that aren't to game
-    # folders or that are to what look like backup game
-    # folders.
-    args[:sport_codes].each do |sport_code|
-      args[:sport_code] = sport_code
-      (Nokogiri::XML(self.get(gameday_league_and_date_url(args)))/"a").reject{|l| l.attribute('href').value[0,4] != "gid_" or l.attribute('href').value[-5,4] == "_bak"}.each do |e|      
-        self.pull_game(e.attribute('href').value.gsub('/',''),args) 
-      end
-    end
 
-    return args[:sport_code]
+  def pull_day(date, sport_code)
+    game_links_on_gameday_date_page(date, sport_code).each do |game_id|
+      pull_game(game_id, date)
+    end
   end
 
   ##
@@ -109,66 +45,95 @@ module Greenmonster::Spider
   # passes arguments like games_folder location on to Spider.pull.
   #
   # Example:
-  #   # Pull all games in MLB in July 2011
-  #   >> Gameday::Spider.pull_days(Date.new(2011,7,1)..Date.new(2011,7,31), {:games_folder => '/Users/geoff/games'})
+  #  # Pull all games in MLB in July 2011
+  #  >> Gameday::Spider.pull_days(Date.new(2011,7,1)..Date.new(2011,7,31), {:games_folder => '/Users/geoff/games'})
   #
   # Arguments:
-  #   range: (Range)
-  #   args: (Hash)
-
-  def self.pull_days(range,args = {})
-    range.each {|day| self.pull_day(args.merge({:date => day}))}
-  end
-    
-    private
-    ##
-    # Return the Gameday URL for a given sport_code
-    # and date. Argument must include :date, but
-    # :sport_code will be assumed to be 'mlb' if not
-    # specified.
-    #
-    # Arguments:
-    #   args: (Hash)
-    #
-    def self.gameday_league_and_date_url(args = {})
-      raise "Date required." unless args[:date]
-      args[:sport_code] ||= 'mlb'
-    
-      "http://gd2.mlb.com/components/game/#{args[:sport_code]}/#{format_date_as_folder(args[:date])}"
+  #  range: (Range)
+  #  args: (Hash)
+
+  def pull_days(range, sport_code)
+    range.each { |date| self.pull_day(date, sport_code) }
+  end
+
+  private
+
+  def get_gameday_date_page(date, sport_code)
+    self.class.get(gameday_date_and_sport_code_url(date, sport_code))
+  end
+
+  def links_on_gameday_date_page(date, sport_code)
+    Nokogiri::XML(get_gameday_date_page(date, sport_code)).search('a').map do |a|
+      a.attribute('href').value
     end
-    
-    ##
-    # Copy XML files from the Gameday severs to your
-    # local machine for parsing and analysis. This method
-    # is used by Spider pull class methods to put files on 
-    # your local machine in a similar layout to the one used by 
-    # Gameday. The paths argument gets built by Spider.pull
-    # during the pull process.
-    #
-    # Arguments: 
-    #   file_name: (String)
-    #   paths: (Hash)
-  
-    def self.copy_gameday_xml (file_name,paths)
-      download = self.get(paths[:mlbGameFolder] + "#{file_name =~ /inning/ ? 'inning/' : ''}" + file_name).body
-      unless download.include?('404 Not Found')
-        open(paths[:localGameFolder] + (file_name =~ /inning/ ? 'inning/' : '') + file_name, 'w') do |file|
-          file.write(download) 
-        end
-      end
+  end
+
+  def game_links_on_gameday_date_page(date, sport_code)
+    links_on_gameday_date_page(date, sport_code).select do |link|
+      link[0,4] == "gid_" && link[-5,4] != "_bak"
+    end
+  end
+
+  def gameday_url_root
+    "http://gd2.mlb.com/components/game/"
+  end
+
+  def gameday_date_and_sport_code_url(date, sport_code)
+    "#{gameday_url_root}#{sport_code}/#{format_date_as_folder(date)}"
+  end
+
+  def gameday_game_url(game_id, date)
+    gameday_url_root + remote_game_path(game_id, date)
+  end
+
+  def remote_game_path(game_id, date)
+    "#{home_sport_code_from_game_id(game_id)}/#{format_date_as_folder(date)}/#{game_id}"
+  end
+
+  def home_sport_code_from_game_id(game_id)
+    game_id[-5,3]
+  end
+
+  def inning_prefix(file_name)
+    if file_name =~ /inning/
+      'inning/'
+    else
+      ''
     end
-  
-    ##
-    # Output a folder format similar to the one used by Gameday.
-    # 
-    # Example:
-    #   >> Spider.format_date_as_folder(Date.new(2011,7,4))
-    #   => "year_2011/month_07/day_04"
-    # 
-    # Arguments:
-    #   date: (Date)
-  
-    def self.format_date_as_folder(date)
-      Greenmonster.format_date_as_folder(date)
+  end
+
+  def remote_file_url(game_id, date, file_name)
+    gameday_game_url(game_id, date) + '/' + inning_prefix(file_name) + '/' + file_name
+  end
+
+  def download_gameday_xml(game_id, date, file_name)
+    self.class.get(remote_file_url(game_id, date, file_name)).body.force_encoding("ISO-8859-1").encode("UTF-8")
+  end
+
+  def local_game_path(game_id, date)
+    Pathname.new(
+      Greenmonster.games_folder +
+      home_sport_code_from_game_id(game_id) +
+      format_date_as_folder(date) +
+      game_id
+    )
+  end
+
+  def copy_gameday_xml(game_id, date, file_name)
+    download = download_gameday_xml(game_id, date, file_name)
+
+    unless download.include?('404 Not Found')
+      open(local_game_path(game_id, date) + inning_prefix(file_name) + file_name, 'w') do |file|
+        file.write(download)
+      end
     end
+  end
+
+  def format_date_as_folder(date)
+    Greenmonster.format_date_as_folder(date)
+  end
+
+  def make_folders_for_game(game_id, date)
+    FileUtils.mkdir_p(local_game_path(game_id, date) + 'inning')
+  end
 end