atp_scraper 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 29b324b13a1ec9af8272f35a9dcddd04062c81a6
4
+ data.tar.gz: 967d3f5ea46058ecb3782d5d335d441c120d6f8c
5
+ SHA512:
6
+ metadata.gz: 6103e7d6c92331867645d0688ba6942ec4487f28b18c581b911ede60288f0a8aa26f318e4778cea8ab809b327f53e0e5cdc80caf8cdc46175bd6c0a55255d854
7
+ data.tar.gz: fdd6848eede81c64f5bf37e0c4cf082ca426b031ff256640c624085d5126e464781cec1051d608168a14d1fa0a0fde3e84c1175685c9db2b10162e92473f78c5
data/.gitignore ADDED
@@ -0,0 +1,10 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /Gemfile.lock
4
+ /_yardoc/
5
+ /coverage/
6
+ /doc/
7
+ /pkg/
8
+ /spec/reports/
9
+ /tmp/
10
+ /vendor/
data/.rubocop.yml ADDED
@@ -0,0 +1,22 @@
1
+ AllCops:
2
+ Exclude:
3
+ - 'Gemfile'
4
+ - 'Rakefile'
5
+ - 'atp_scraper.gemspec'
6
+ - 'test/**/*'
7
+ - 'vendor/**/*'
8
+
9
+ Metrics/LineLength:
10
+ Max: 100
11
+
12
+ Metrics/MethodLength:
13
+ Max: 20
14
+
15
+ Style/StringLiterals:
16
+ Enabled: false
17
+
18
+ Style/MultilineOperationIndentation:
19
+ EnforcedStyle: indented
20
+
21
+ Style/MutableConstant:
22
+ Enabled: false
data/.travis.yml ADDED
@@ -0,0 +1,4 @@
1
+ language: ruby
2
+ rvm:
3
+ - 2.1.5
4
+ before_install: gem install bundler -v 1.10.2
@@ -0,0 +1,13 @@
1
+ # Contributor Code of Conduct
2
+
3
+ As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
4
+
5
+ We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion.
6
+
7
+ Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.
8
+
9
+ Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.
10
+
11
+ Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.
12
+
13
+ This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in atp_scraper.gemspec
4
+ gemspec
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2016 mosuke5
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,71 @@
1
+ [![Build Status](https://travis-ci.org/mosuke5/atp_scraper.svg?branch=master)](https://travis-ci.org/mosuke5/atp_scraper)
2
+ # AtpScraper
3
+
4
+ Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/atp_scraper`. To experiment with that code, run `bin/console` for an interactive prompt.
5
+
6
+ TODO: Delete this and the text above, and describe your gem
7
+
8
+ ## Installation
9
+
10
+ Add this line to your application's Gemfile:
11
+
12
+ ```ruby
13
+ gem 'atp_scraper'
14
+ ```
15
+
16
+ And then execute:
17
+
18
+ $ bundle
19
+
20
+ Or install it yourself as:
21
+
22
+ $ gem install atp_scraper
23
+
24
+ ## Usage
25
+ ```ruby
26
+ require "atp_scraper"
27
+
28
+ # Get Singles Ranking
29
+ AtpScraper::Ranking.get
30
+ # Response Json
31
+ # {
32
+ # player_name: "Rafael Nadal",
33
+ # player_url_name: "rafael-nadal",
34
+ # player_id: "n409"
35
+ # }
36
+
37
+ # Get Player Activity. For Example Rafael Nadal's activity in 2016
38
+ AtpScraper::Activity.get("n409", 2016)
39
+ # Response Json
40
+ # {
41
+ # year: 2016,
42
+ # player_name: "Rafael Nadal",
43
+ # player_rank: 5,
44
+ # opponent_name: "Fernand Verdasco",
45
+ # opponent_rank: 45,
46
+ # round: "Round of 128",
47
+ # score: "673 64 63 674 26",
48
+ # win_loss: "L",
49
+ # tournament_name: "Australian Open",
50
+ # tournament_location: "Melbourne, Australia",
51
+ # tournament_start_date: "2016.01.18",
52
+ # tournament_end_date: "2016.01.31",
53
+ # tournament_surface: "OurdoorHard"
54
+ # }
55
+ ```
56
+
57
+ ## Development
58
+
59
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
60
+
61
+ To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
62
+
63
+ ## Contributing
64
+
65
+ Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/atp_scraper. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](contributor-covenant.org) code of conduct.
66
+
67
+
68
+ ## License
69
+
70
+ The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
71
+
data/Rakefile ADDED
@@ -0,0 +1,10 @@
1
+ require "bundler/gem_tasks"
2
+ require "rake/testtask"
3
+
4
+ Rake::TestTask.new(:test) do |t|
5
+ t.libs << "test"
6
+ t.libs << "lib"
7
+ t.test_files = FileList['test/**/test_*.rb']
8
+ end
9
+
10
+ task :default => :test
@@ -0,0 +1,30 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'atp_scraper/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = "atp_scraper"
8
+ spec.version = AtpScraper::VERSION
9
+ spec.authors = ["mosuke5"]
10
+ spec.email = ["w.vamos603@gmail.com"]
11
+
12
+ spec.summary = %q{Scraping tool from ATP tennis site.}
13
+ spec.description = %q{atp_scraper is a tool scraping tennis data from atpworldtour.com}
14
+ spec.homepage = "https://github.com/mosuke5/atp_scraper"
15
+ spec.license = "MIT"
16
+
17
+ spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
18
+ spec.bindir = "exe"
19
+ spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
20
+ spec.require_paths = ["lib"]
21
+
22
+ spec.add_dependency "nokogiri"
23
+ spec.add_development_dependency "bundler", "~> 1.10"
24
+ spec.add_development_dependency "rake", "~> 10.0"
25
+ spec.add_development_dependency "test-unit"
26
+ spec.add_development_dependency "simplecov"
27
+ spec.add_development_dependency "pry"
28
+ spec.add_development_dependency "rb-readline"
29
+ spec.add_development_dependency "rubocop"
30
+ end
data/bin/console ADDED
@@ -0,0 +1,10 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require "bundler/setup"
4
+ require "atp_scraper"
5
+
6
+ # You can add fixtures and/or initialization code here to make experimenting
7
+ # with your gem easier. You can also use a different console, if you like.
8
+
9
+ require "pry"
10
+ Pry.start
data/bin/setup ADDED
@@ -0,0 +1,7 @@
1
+ #!/bin/bash
2
+ set -euo pipefail
3
+ IFS=$'\n\t'
4
+
5
+ bundle install
6
+
7
+ # Do any other automated setup that you need to do here
@@ -0,0 +1,133 @@
1
+ ## Page Structure
2
+ # activity_doc(whole)
3
+ # tournament_doc
4
+ # record_doc
5
+ # record_doc
6
+ # record_doc
7
+ # record_doc
8
+ # tournament_doc
9
+ # record_doc
10
+ # record_doc
11
+ # record_doc
12
+ ###################
13
+
14
+ module AtpScraper
15
+ # Scrape activity data
16
+ class Activity
17
+ class << self
18
+ def get(player_id, year)
19
+ request_uri = build_uri(player_id, year)
20
+ activity_doc = AtpScraper::Html.get_and_parse(request_uri)
21
+ pickup_activity_data(activity_doc)
22
+ end
23
+
24
+ def build_uri(player_id, year)
25
+ "/players/anything/#{player_id}/player-activity?year=#{year}"
26
+ end
27
+
28
+ def pickup_activity_data(activity_doc)
29
+ result = []
30
+ player = {}
31
+ player[:name] = pickup_player_name(activity_doc)
32
+
33
+ search_tournaments_doc(activity_doc).each do |tournament_doc|
34
+ tournament = pickup_tournament_info(tournament_doc)
35
+ player[:rank] = pickup_player_rank(tournament[:caption])
36
+ search_records_doc(tournament_doc).each do |record_doc|
37
+ record = pickup_record(record_doc)
38
+ record_hash = create_record(record, player, tournament)
39
+ result.push(record_hash)
40
+ end
41
+ end
42
+ result
43
+ end
44
+
45
+ def search_tournaments_doc(activity_doc)
46
+ activity_doc.css(".activity-tournament-table")
47
+ end
48
+
49
+ def search_records_doc(tournament_doc)
50
+ tournament_doc.css(".mega-table tbody tr")
51
+ end
52
+
53
+ def create_record(record, player, tournament)
54
+ {
55
+ year: tournament[:year],
56
+ player_name: player[:name],
57
+ player_rank: player[:rank],
58
+ opponent_name: record[:opponent_name],
59
+ opponent_rank: record[:opponent_rank],
60
+ round: record[:round],
61
+ score: record[:score],
62
+ win_loss: record[:win_loss],
63
+ tournament_name: tournament[:name],
64
+ tournament_location: tournament[:location],
65
+ tournament_start_date: tournament[:date][:start],
66
+ tournament_end_date: tournament[:date][:end],
67
+ tournament_surface: tournament[:surface]
68
+ }
69
+ end
70
+
71
+ def pickup_player_name(activity_doc)
72
+ activity_doc
73
+ .css("meta[property=\"pageTransitionTitle\"]")
74
+ .attr("content").value
75
+ end
76
+
77
+ def pickup_record(record_doc)
78
+ result = {}
79
+ record_doc.css("td").each_with_index do |td, n|
80
+ record_content = td.content.strip
81
+ case n
82
+ when 0 then
83
+ result[:round] = record_content
84
+ when 1 then
85
+ result[:opponent_rank] = record_content
86
+ when 2 then
87
+ result[:opponent_name] = record_content
88
+ when 3 then
89
+ result[:win_loss] = record_content
90
+ when 4 then
91
+ result[:score] = record_content
92
+ end
93
+ end
94
+ result
95
+ end
96
+
97
+ def pickup_tournament_info(tournament_doc)
98
+ tournament_date = pickup_text(tournament_doc, ".tourney-dates")
99
+ {
100
+ name: pickup_text(tournament_doc, ".tourney-title"),
101
+ location: pickup_text(tournament_doc, ".tourney-location"),
102
+ date: divide_tournament_date(tournament_date),
103
+ year: tournament_date[0, 4],
104
+ caption: pickup_text(tournament_doc, ".activity-tournament-caption"),
105
+ surface: pickup_surface(tournament_doc)
106
+ }
107
+ end
108
+
109
+ def pickup_player_rank(tournament_caption)
110
+ rank = tournament_caption.match(/ATP Ranking:(.+), Prize/)
111
+ rank[1].strip
112
+ end
113
+
114
+ # Before: String "2011.01.03 - 2011.01.08"
115
+ # After: Hash { start: 2011.01.03, end: 2011.01.08 }
116
+ def divide_tournament_date(date)
117
+ date = date.split('-').map(&:strip)
118
+ { start: date[0], end: date[1] }
119
+ end
120
+
121
+ def pickup_text(doc, selector)
122
+ doc.css(selector).first.content.strip
123
+ end
124
+
125
+ def pickup_surface(tournament_doc)
126
+ tournament_doc
127
+ .css(".tourney-details")[1]
128
+ .css(".item-details")
129
+ .first.content.gsub(/\t|\s/, "")
130
+ end
131
+ end
132
+ end
133
+ end
@@ -0,0 +1,25 @@
1
+ require 'open-uri'
2
+ require 'nokogiri'
3
+ module AtpScraper
4
+ # Get and parse html from atpworldtour.com
5
+ class Html
6
+ BASE = "http://www.atpworldtour.com"
7
+ def self.get_and_parse(uri)
8
+ html = get(uri)
9
+ parse(html[:html], html[:charset])
10
+ end
11
+
12
+ def self.get(uri)
13
+ charset = nil
14
+ html = open(BASE + uri) do |f|
15
+ charset = f.charset
16
+ f.read
17
+ end
18
+ { html: html, charset: charset }
19
+ end
20
+
21
+ def self.parse(html, charset)
22
+ Nokogiri::HTML.parse(html, nil, charset)
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,58 @@
1
+ ## Structure Rnaking Page
2
+ # ranking_doc
3
+ # no1. player_doc
4
+ # no2. player_doc
5
+ # no3. player_doc
6
+ # no4. player_doc
7
+ #################
8
+
9
+ module AtpScraper
10
+ # Scrape ranking data
11
+ class Ranking
12
+ class << self
13
+ def get
14
+ request_uri = "/en/rankings/singles"
15
+ ranking_doc = AtpScraper::Html.get_and_parse(request_uri)
16
+ pickup_ranking_data(ranking_doc)
17
+ end
18
+
19
+ def pickup_ranking_data(ranking_doc)
20
+ result = []
21
+ search_player_doc(ranking_doc).each do |player_doc|
22
+ result.push(pickup_player_data(player_doc))
23
+ end
24
+ result
25
+ end
26
+
27
+ def search_player_doc(ranking_doc)
28
+ ranking_doc.css(".mega-table tbody tr")
29
+ end
30
+
31
+ def pickup_player_data(player_doc)
32
+ url = pickup_player_url(player_doc)
33
+ {
34
+ player_name: pickup_player_name(player_doc),
35
+ player_url_name: get_url_name(url),
36
+ player_id: get_url_id(url)
37
+ }
38
+ end
39
+
40
+ def pickup_player_name(player_doc)
41
+ player_doc.css(".player-cell").first.content.strip
42
+ end
43
+
44
+ def pickup_player_url(player_doc)
45
+ player_doc.css(".player-cell a").attr("href").value
46
+ end
47
+
48
+ # url => "/en/players/rafael-nadal/n409/overview"
49
+ def get_url_name(url)
50
+ url.split("/")[3]
51
+ end
52
+
53
+ def get_url_id(url)
54
+ url.split("/")[4]
55
+ end
56
+ end
57
+ end
58
+ end
@@ -0,0 +1,3 @@
1
+ module AtpScraper
2
+ VERSION = "0.1.0"
3
+ end
@@ -0,0 +1,4 @@
1
+ require "atp_scraper/version"
2
+ require "atp_scraper/activity"
3
+ require "atp_scraper/ranking"
4
+ require "atp_scraper/html"
metadata ADDED
@@ -0,0 +1,172 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: atp_scraper
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - mosuke5
8
+ autorequire:
9
+ bindir: exe
10
+ cert_chain: []
11
+ date: 2016-02-15 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: nokogiri
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: bundler
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '1.10'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '1.10'
41
+ - !ruby/object:Gem::Dependency
42
+ name: rake
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '10.0'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '10.0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: test-unit
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ - !ruby/object:Gem::Dependency
70
+ name: simplecov
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - ">="
74
+ - !ruby/object:Gem::Version
75
+ version: '0'
76
+ type: :development
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - ">="
81
+ - !ruby/object:Gem::Version
82
+ version: '0'
83
+ - !ruby/object:Gem::Dependency
84
+ name: pry
85
+ requirement: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - ">="
88
+ - !ruby/object:Gem::Version
89
+ version: '0'
90
+ type: :development
91
+ prerelease: false
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - ">="
95
+ - !ruby/object:Gem::Version
96
+ version: '0'
97
+ - !ruby/object:Gem::Dependency
98
+ name: rb-readline
99
+ requirement: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - ">="
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ type: :development
105
+ prerelease: false
106
+ version_requirements: !ruby/object:Gem::Requirement
107
+ requirements:
108
+ - - ">="
109
+ - !ruby/object:Gem::Version
110
+ version: '0'
111
+ - !ruby/object:Gem::Dependency
112
+ name: rubocop
113
+ requirement: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - ">="
116
+ - !ruby/object:Gem::Version
117
+ version: '0'
118
+ type: :development
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - ">="
123
+ - !ruby/object:Gem::Version
124
+ version: '0'
125
+ description: atp_scraper is a tool scraping tennis data from atpworldtour.com
126
+ email:
127
+ - w.vamos603@gmail.com
128
+ executables: []
129
+ extensions: []
130
+ extra_rdoc_files: []
131
+ files:
132
+ - ".gitignore"
133
+ - ".rubocop.yml"
134
+ - ".travis.yml"
135
+ - CODE_OF_CONDUCT.md
136
+ - Gemfile
137
+ - LICENSE.txt
138
+ - README.md
139
+ - Rakefile
140
+ - atp_scraper.gemspec
141
+ - bin/console
142
+ - bin/setup
143
+ - lib/atp_scraper.rb
144
+ - lib/atp_scraper/activity.rb
145
+ - lib/atp_scraper/html.rb
146
+ - lib/atp_scraper/ranking.rb
147
+ - lib/atp_scraper/version.rb
148
+ homepage: https://github.com/mosuke5/atp_scraper
149
+ licenses:
150
+ - MIT
151
+ metadata: {}
152
+ post_install_message:
153
+ rdoc_options: []
154
+ require_paths:
155
+ - lib
156
+ required_ruby_version: !ruby/object:Gem::Requirement
157
+ requirements:
158
+ - - ">="
159
+ - !ruby/object:Gem::Version
160
+ version: '0'
161
+ required_rubygems_version: !ruby/object:Gem::Requirement
162
+ requirements:
163
+ - - ">="
164
+ - !ruby/object:Gem::Version
165
+ version: '0'
166
+ requirements: []
167
+ rubyforge_project:
168
+ rubygems_version: 2.5.1
169
+ signing_key:
170
+ specification_version: 4
171
+ summary: Scraping tool from ATP tennis site.
172
+ test_files: []