sportdb-importers 1.1.1 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +68 -11
- data/Rakefile +1 -1
- data/lib/sportdb/importers.rb +5 -7
- data/lib/sportdb/importers/event.rb +5 -3
- data/lib/sportdb/importers/match.rb +96 -96
- data/lib/sportdb/importers/version.rb +2 -4
- data/test/test_club.rb +162 -162
- data/test/test_import.rb +55 -55
- data/test/test_version.rb +22 -22
- metadata +12 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 132390776871652a1bed3e4edc1760dff818fe9c
|
|
4
|
+
data.tar.gz: a6a0c56499e48a671c4711837bd5cc668f6a3212
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f31c12e310f79be4d6de7ebf462bbe04e754718f1af03bb049b80c1abdc3e54c01b4dd82164219b777fe32606fcb4dcbc73423f115fd4d1df3791585d70ae6ee
|
|
7
|
+
data.tar.gz: b0a3fc94b27d60c4037fd5deb764a7457a43da50fe3f0ab283364f642b93440f563317e43aac5f33efe129883d90e41ea776f553550dfdbae836a9b1a12da158
|
data/README.md
CHANGED
|
@@ -30,17 +30,17 @@ SportDb.create_all ## build database schema (tables, indexes, etc.)
|
|
|
30
30
|
Let's use the public domain football.csv datasets for England (see [`footballcsv/england`](https://github.com/footballcsv/england)), as an example:
|
|
31
31
|
|
|
32
32
|
```
|
|
33
|
-
Round, Date,
|
|
34
|
-
1,
|
|
35
|
-
1,
|
|
36
|
-
1,
|
|
37
|
-
1,
|
|
38
|
-
1,
|
|
39
|
-
1,
|
|
40
|
-
1,
|
|
41
|
-
1,
|
|
42
|
-
1,
|
|
43
|
-
1,
|
|
33
|
+
Round, Date, Team 1, FT, HT, Team 2
|
|
34
|
+
1, Fri Aug 9 2019, Liverpool FC, 4-1, 4-0, Norwich City FC
|
|
35
|
+
1, Sat Aug 10 2019, West Ham United FC, 0-5, 0-1, Manchester City FC
|
|
36
|
+
1, Sat Aug 10 2019, AFC Bournemouth, 1-1, 0-0, Sheffield United FC
|
|
37
|
+
1, Sat Aug 10 2019, Burnley FC, 3-0, 0-0, Southampton FC
|
|
38
|
+
1, Sat Aug 10 2019, Crystal Palace FC, 0-0, 0-0, Everton FC
|
|
39
|
+
1, Sat Aug 10 2019, Watford FC, 0-3, 0-1, Brighton & Hove Albion FC
|
|
40
|
+
1, Sat Aug 10 2019, Tottenham Hotspur FC, 3-1, 0-1, Aston Villa FC
|
|
41
|
+
1, Sun Aug 11 2019, Leicester City FC, 0-0, 0-0, Wolverhampton Wanderers FC
|
|
42
|
+
1, Sun Aug 11 2019, Newcastle United FC, 0-1, 0-0, Arsenal FC
|
|
43
|
+
1, Sun Aug 11 2019, Manchester United FC, 4-0, 1-0, Chelsea FC
|
|
44
44
|
...
|
|
45
45
|
```
|
|
46
46
|
(Source: [england/2019-20/eng.1.csv](https://github.com/footballcsv/england/blob/master/2010s/2019-20/eng.1.csv))
|
|
@@ -76,6 +76,63 @@ SportDb.read_csv( './england.zip' )
|
|
|
76
76
|
That's it.
|
|
77
77
|
|
|
78
78
|
|
|
79
|
+
## Frequently Asked Questions (FAQ) and Answers
|
|
80
|
+
|
|
81
|
+
### Q: What CSV formats can I use?
|
|
82
|
+
|
|
83
|
+
For now the importers support two flavors.
|
|
84
|
+
|
|
85
|
+
Alternative 1) One league and season per datafile and
|
|
86
|
+
the basename (e.g. `eng.1` ) holds the league code
|
|
87
|
+
and the directory (e.g. `2019-20`) the season.
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
Matchday, Date, Time, Team 1, FT, Team 2
|
|
91
|
+
1, Fri Aug 9 2019, 20:00, Liverpool, 4-1, Norwich City
|
|
92
|
+
1, Sat Aug 10 2019, 12:30, West Ham, 0-5, Manchester City
|
|
93
|
+
...
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Alternative 2) Any or many leagues or seasons per datafile,
|
|
97
|
+
for example, week by week (see [`/updates`](https://github.com/footballcsv/cache.updates))
|
|
98
|
+
or year by year (see [`/internationals`](https://github.com/footballcsv/cache.internationals)).
|
|
99
|
+
|
|
100
|
+
```
|
|
101
|
+
Date, League, Team 1, FT, Team 2
|
|
102
|
+
Wed Jun 10 2020, DE 3, SpVgg Unterhaching, 1-3, Eintracht Braunschweig
|
|
103
|
+
Thu Jun 11 2020, AT 2, FC Blau-Weiß Linz, 1-2, Austria Klagenfurt
|
|
104
|
+
Thu Jun 11 2020, ES 1, Sevilla FC, 2-0, Real Betis
|
|
105
|
+
...
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
Note: For now the convention is that the datafile basename
|
|
109
|
+
MUST be all numbers, that is, `0` to `9` (plus `-` or `_`) e.g.
|
|
110
|
+
`01` (as in `2020/01.csv`) or `2020` (as in `2000s/2020.csv`).
|
|
111
|
+
|
|
112
|
+
|
|
113
|
+
### Q: What codes or names for league & cups can I use?
|
|
114
|
+
|
|
115
|
+
The importers ship with hundreds of zero-config preconfigured
|
|
116
|
+
code and names for leagues & cups.
|
|
117
|
+
See the [/leagues](https://github.com/openfootball/leagues) datasets for all builtin national and international football club leagues & cups from around the world.
|
|
118
|
+
|
|
119
|
+
Or to query in ruby try:
|
|
120
|
+
|
|
121
|
+
``` ruby
|
|
122
|
+
require `sportdb/config`
|
|
123
|
+
|
|
124
|
+
LEAGUES = SportDb::Import.catalog.leagues
|
|
125
|
+
|
|
126
|
+
LEAGUES.find( 'ENG 1' ) #=> Premier League › England
|
|
127
|
+
LEAGUES.find( 'EPL' ) #=> Premier League › England
|
|
128
|
+
LEAGUES.find( 'ENG 2' ) #=> Championship › England
|
|
129
|
+
LEAGUES.find( 'ENG CS' ) #=> Championship › England
|
|
130
|
+
LEAGUES.find( 'ES' ) #=> Primera División › Spain
|
|
131
|
+
LEAUGES.find( 'ESP 1') #=> Primera División › Spain
|
|
132
|
+
...
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
|
|
79
136
|
|
|
80
137
|
## License
|
|
81
138
|
|
data/Rakefile
CHANGED
|
@@ -8,7 +8,7 @@ Hoe.spec 'sportdb-importers' do
|
|
|
8
8
|
self.summary = "sportdb-importers - tools 'n' scripts for importing sports (football) data in alternate (text) formats incl. comma-separated values (csv) format"
|
|
9
9
|
self.description = summary
|
|
10
10
|
|
|
11
|
-
self.urls =
|
|
11
|
+
self.urls = { home: 'https://github.com/sportdb/sport.db' }
|
|
12
12
|
|
|
13
13
|
self.author = 'Gerald Bauer'
|
|
14
14
|
self.email = 'opensport@googlegroups.com'
|
data/lib/sportdb/importers.rb
CHANGED
|
@@ -1,5 +1,3 @@
|
|
|
1
|
-
# encoding: utf-8
|
|
2
|
-
|
|
3
1
|
|
|
4
2
|
## 3rd party gemss
|
|
5
3
|
require 'sportdb/sync'
|
|
@@ -53,7 +51,7 @@ def self.handle_csv( source, start: nil )
|
|
|
53
51
|
## filter.skip? filter.include? ( season_sason_key )?
|
|
54
52
|
## fiteer.before?( season_key ) etc.
|
|
55
53
|
## find some good method names!!!!
|
|
56
|
-
season_start = start ?
|
|
54
|
+
season_start = start ? Season.parse( start ) : nil
|
|
57
55
|
|
|
58
56
|
if source.is_a?( Datafile::DirPackage::Entry) ||
|
|
59
57
|
source.is_a?( Datafile::ZipPackage::Entry)
|
|
@@ -72,10 +70,10 @@ def self.handle_csv( source, start: nil )
|
|
|
72
70
|
|
|
73
71
|
## todo/fix: check if season_key is proper season - e.g. matches pattern !!!!
|
|
74
72
|
season_q = File.basename( File.dirname( entry.name ))
|
|
75
|
-
season =
|
|
73
|
+
season = Season.parse( season_q ) ## normalize season
|
|
76
74
|
season_key = season.key
|
|
77
75
|
|
|
78
|
-
if season_start && season_start
|
|
76
|
+
if season_start && season_start > season
|
|
79
77
|
## skip if start season before this season
|
|
80
78
|
else
|
|
81
79
|
pp [entry.name, season_key, league_key]
|
|
@@ -106,10 +104,10 @@ def self.handle_csv( source, start: nil )
|
|
|
106
104
|
league_key = basename
|
|
107
105
|
|
|
108
106
|
season_q = File.basename( File.dirname( full_path ) )
|
|
109
|
-
season =
|
|
107
|
+
season = Season.parse( season_q ) ## normalize season
|
|
110
108
|
season_key = season.key
|
|
111
109
|
|
|
112
|
-
if season_start && season_start
|
|
110
|
+
if season_start && season_start > season
|
|
113
111
|
## skip if start season before this season
|
|
114
112
|
else
|
|
115
113
|
## todo/fix: check if season_key is proper season - e.g. matches pattern !!!!
|
|
@@ -6,13 +6,15 @@ class CsvEventImporter
|
|
|
6
6
|
def self.read( path, league:, season:,
|
|
7
7
|
headers: nil )
|
|
8
8
|
txt = File.open( path, 'r:utf-8' ) {|f| f.read }
|
|
9
|
-
parse( txt, league:
|
|
9
|
+
parse( txt, league: league,
|
|
10
|
+
season: season,
|
|
10
11
|
headers: headers )
|
|
11
12
|
end
|
|
12
13
|
|
|
13
14
|
def self.parse( txt, league:, season:,
|
|
14
15
|
headers: nil )
|
|
15
|
-
new( txt, league:
|
|
16
|
+
new( txt, league: league,
|
|
17
|
+
season: season,
|
|
16
18
|
headers: headers ).parse
|
|
17
19
|
end
|
|
18
20
|
|
|
@@ -26,7 +28,7 @@ class CsvEventImporter
|
|
|
26
28
|
|
|
27
29
|
## try mapping of league here - why? why not?
|
|
28
30
|
@league = Import.catalog.leagues.find!( league )
|
|
29
|
-
@season =
|
|
31
|
+
@season = Season.parse( season )
|
|
30
32
|
end
|
|
31
33
|
|
|
32
34
|
|
|
@@ -1,96 +1,96 @@
|
|
|
1
|
-
|
|
2
|
-
|
|
3
|
-
module SportDb
|
|
4
|
-
class CsvMatchImporter
|
|
5
|
-
|
|
6
|
-
def self.read( path, headers: nil )
|
|
7
|
-
txt = File.open( path, 'r:utf-8' ) {|f| f.read }
|
|
8
|
-
parse( txt, headers: headers )
|
|
9
|
-
end
|
|
10
|
-
|
|
11
|
-
def self.parse( txt, headers: nil )
|
|
12
|
-
new( txt, headers: headers ).parse
|
|
13
|
-
end
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
def initialize( txt, headers: nil )
|
|
17
|
-
@txt = txt
|
|
18
|
-
@headers = headers
|
|
19
|
-
end
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
def parse
|
|
23
|
-
## todo/fix: add headers options (pass throughto CsvMatchReader)
|
|
24
|
-
## add filters too why? why not?
|
|
25
|
-
|
|
26
|
-
## todo/fix:
|
|
27
|
-
## add normalize: false/mapping: false flag for NOT mapping club/team names
|
|
28
|
-
## make normalize: false the default, anyways - why? why not?
|
|
29
|
-
opts = {}
|
|
30
|
-
opts[:headers] = @headers if @headers
|
|
31
|
-
|
|
32
|
-
matches = CsvMatchParser.parse( @txt, **opts )
|
|
33
|
-
|
|
34
|
-
matches.each do |match|
|
|
35
|
-
league = Import.catalog.leagues.find!( match.league )
|
|
36
|
-
# pp league
|
|
37
|
-
|
|
38
|
-
team1 = Import.catalog.teams.find_by!( name: match.team1, league: league )
|
|
39
|
-
team2 = Import.catalog.teams.find_by!( name: match.team2, league: league )
|
|
40
|
-
|
|
41
|
-
date = Date.strptime( match.date, '%Y-%m-%d' )
|
|
42
|
-
## quick hack - for now always use 2019/20 style season
|
|
43
|
-
## fix!!! - use league to find proper season e.g. 2019 or 2019/20 etc.
|
|
44
|
-
|
|
45
|
-
start_year = if date.month >= 7
|
|
46
|
-
date.year
|
|
47
|
-
else
|
|
48
|
-
date.year-1
|
|
49
|
-
end
|
|
50
|
-
|
|
51
|
-
## note: for now always assume 2019/20 season
|
|
52
|
-
season =
|
|
53
|
-
# pp season
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
event_rec = Sync::Event.find_or_create_by( league: league,
|
|
57
|
-
season: season )
|
|
58
|
-
|
|
59
|
-
team1_rec = Sync::Team.find_or_create( team1 )
|
|
60
|
-
team2_rec = Sync::Team.find_or_create( team2 )
|
|
61
|
-
|
|
62
|
-
## warn about duplicates?
|
|
63
|
-
## note: for now only allow one (unique) match pair per team
|
|
64
|
-
match_recs = Model::Match.where( event_id: event_rec.id,
|
|
65
|
-
team1_id: team1_rec.id,
|
|
66
|
-
team2_id: team2_rec.id ).to_a
|
|
67
|
-
if match_recs.size > 0
|
|
68
|
-
puts "!! #{match_recs.size} duplicate match record(s) found:"
|
|
69
|
-
pp match_recs
|
|
70
|
-
exit 1
|
|
71
|
-
end
|
|
72
|
-
|
|
73
|
-
## find last pos - check if it can be nil? yes, is nil if no records found
|
|
74
|
-
max_pos = Model::Match.where( event_id: event_rec.id ).maximum( 'pos' )
|
|
75
|
-
max_pos = max_pos ? max_pos+1 : 1
|
|
76
|
-
|
|
77
|
-
rec = Model::Match.create!(
|
|
78
|
-
event_id: event_rec.id,
|
|
79
|
-
team1_id: team1_rec.id,
|
|
80
|
-
team2_id: team2_rec.id,
|
|
81
|
-
## round_id: round_rec.id, -- note: now optional
|
|
82
|
-
pos: max_pos,
|
|
83
|
-
date: date.to_date,
|
|
84
|
-
score1: match.score1,
|
|
85
|
-
score2: match.score2,
|
|
86
|
-
score1i: match.score1i,
|
|
87
|
-
score2i: match.score2i )
|
|
88
|
-
|
|
89
|
-
## todo/fix:
|
|
90
|
-
## check if event includes teams?
|
|
91
|
-
## if not (auto-)add teams to event.teams !!!!!!!
|
|
92
|
-
end
|
|
93
|
-
end # method parse
|
|
94
|
-
|
|
95
|
-
end # class CsvEventImporter
|
|
96
|
-
end # module SportDb
|
|
1
|
+
|
|
2
|
+
|
|
3
|
+
module SportDb
|
|
4
|
+
class CsvMatchImporter
|
|
5
|
+
|
|
6
|
+
def self.read( path, headers: nil )
|
|
7
|
+
txt = File.open( path, 'r:utf-8' ) {|f| f.read }
|
|
8
|
+
parse( txt, headers: headers )
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def self.parse( txt, headers: nil )
|
|
12
|
+
new( txt, headers: headers ).parse
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
def initialize( txt, headers: nil )
|
|
17
|
+
@txt = txt
|
|
18
|
+
@headers = headers
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
def parse
|
|
23
|
+
## todo/fix: add headers options (pass throughto CsvMatchReader)
|
|
24
|
+
## add filters too why? why not?
|
|
25
|
+
|
|
26
|
+
## todo/fix:
|
|
27
|
+
## add normalize: false/mapping: false flag for NOT mapping club/team names
|
|
28
|
+
## make normalize: false the default, anyways - why? why not?
|
|
29
|
+
opts = {}
|
|
30
|
+
opts[:headers] = @headers if @headers
|
|
31
|
+
|
|
32
|
+
matches = CsvMatchParser.parse( @txt, **opts )
|
|
33
|
+
|
|
34
|
+
matches.each do |match|
|
|
35
|
+
league = Import.catalog.leagues.find!( match.league )
|
|
36
|
+
# pp league
|
|
37
|
+
|
|
38
|
+
team1 = Import.catalog.teams.find_by!( name: match.team1, league: league )
|
|
39
|
+
team2 = Import.catalog.teams.find_by!( name: match.team2, league: league )
|
|
40
|
+
|
|
41
|
+
date = Date.strptime( match.date, '%Y-%m-%d' )
|
|
42
|
+
## quick hack - for now always use 2019/20 style season
|
|
43
|
+
## fix!!! - use league to find proper season e.g. 2019 or 2019/20 etc.
|
|
44
|
+
|
|
45
|
+
start_year = if date.month >= 7
|
|
46
|
+
date.year
|
|
47
|
+
else
|
|
48
|
+
date.year-1
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
## note: for now always assume 2019/20 season
|
|
52
|
+
season = Season.parse( '%d/%d' % [start_year, (start_year+1)%100] )
|
|
53
|
+
# pp season
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
event_rec = Sync::Event.find_or_create_by( league: league,
|
|
57
|
+
season: season )
|
|
58
|
+
|
|
59
|
+
team1_rec = Sync::Team.find_or_create( team1 )
|
|
60
|
+
team2_rec = Sync::Team.find_or_create( team2 )
|
|
61
|
+
|
|
62
|
+
## warn about duplicates?
|
|
63
|
+
## note: for now only allow one (unique) match pair per team
|
|
64
|
+
match_recs = Model::Match.where( event_id: event_rec.id,
|
|
65
|
+
team1_id: team1_rec.id,
|
|
66
|
+
team2_id: team2_rec.id ).to_a
|
|
67
|
+
if match_recs.size > 0
|
|
68
|
+
puts "!! #{match_recs.size} duplicate match record(s) found:"
|
|
69
|
+
pp match_recs
|
|
70
|
+
exit 1
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
## find last pos - check if it can be nil? yes, is nil if no records found
|
|
74
|
+
max_pos = Model::Match.where( event_id: event_rec.id ).maximum( 'pos' )
|
|
75
|
+
max_pos = max_pos ? max_pos+1 : 1
|
|
76
|
+
|
|
77
|
+
rec = Model::Match.create!(
|
|
78
|
+
event_id: event_rec.id,
|
|
79
|
+
team1_id: team1_rec.id,
|
|
80
|
+
team2_id: team2_rec.id,
|
|
81
|
+
## round_id: round_rec.id, -- note: now optional
|
|
82
|
+
pos: max_pos,
|
|
83
|
+
date: date.to_date,
|
|
84
|
+
score1: match.score1,
|
|
85
|
+
score2: match.score2,
|
|
86
|
+
score1i: match.score1i,
|
|
87
|
+
score2i: match.score2i )
|
|
88
|
+
|
|
89
|
+
## todo/fix:
|
|
90
|
+
## check if event includes teams?
|
|
91
|
+
## if not (auto-)add teams to event.teams !!!!!!!
|
|
92
|
+
end
|
|
93
|
+
end # method parse
|
|
94
|
+
|
|
95
|
+
end # class CsvEventImporter
|
|
96
|
+
end # module SportDb
|
|
@@ -1,13 +1,11 @@
|
|
|
1
|
-
# encoding: utf-8
|
|
2
|
-
|
|
3
1
|
|
|
4
2
|
module SportDb
|
|
5
3
|
module Module
|
|
6
4
|
module Importers
|
|
7
5
|
|
|
8
6
|
MAJOR = 1 ## todo: namespace inside version or something - why? why not??
|
|
9
|
-
MINOR =
|
|
10
|
-
PATCH =
|
|
7
|
+
MINOR = 2
|
|
8
|
+
PATCH = 0
|
|
11
9
|
VERSION = [MAJOR,MINOR,PATCH].join('.')
|
|
12
10
|
|
|
13
11
|
def self.version
|
data/test/test_club.rb
CHANGED
|
@@ -1,162 +1,162 @@
|
|
|
1
|
-
###
|
|
2
|
-
# to run use
|
|
3
|
-
# ruby -I ./lib -I ./test test/test_club.rb
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
require 'helper'
|
|
7
|
-
|
|
8
|
-
class TestClub < MiniTest::Test
|
|
9
|
-
|
|
10
|
-
def test_eng_i
|
|
11
|
-
## todo/fix:
|
|
12
|
-
## add guest1_country_id (optional) to league (e.g. wales for english premier leaguage)
|
|
13
|
-
## (e.g. canada for mls)
|
|
14
|
-
## (e.g. liechtenstein for swiss superleague) etc.
|
|
15
|
-
## why? why not?
|
|
16
|
-
|
|
17
|
-
## fetch English Premiere League
|
|
18
|
-
league = LEAGUES.find!( 'ENG' )
|
|
19
|
-
|
|
20
|
-
team_names = [
|
|
21
|
-
'Manchester City',
|
|
22
|
-
'Arsenal',
|
|
23
|
-
'Liverpool',
|
|
24
|
-
'Cardiff',
|
|
25
|
-
]
|
|
26
|
-
|
|
27
|
-
recs = TEAMS.find_by!( name: team_names,
|
|
28
|
-
league: league )
|
|
29
|
-
|
|
30
|
-
assert_equal 4, recs.size
|
|
31
|
-
|
|
32
|
-
assert_equal 'Manchester City FC', recs[0].name
|
|
33
|
-
## assert_equal 'Manchester', recs[0].city.name
|
|
34
|
-
assert_equal 'England', recs[0].country.name
|
|
35
|
-
|
|
36
|
-
assert_equal 'Arsenal FC', recs[1].name
|
|
37
|
-
## assert_equal 'London', recs[1].city.name
|
|
38
|
-
assert_equal 'England', recs[1].country.name
|
|
39
|
-
|
|
40
|
-
assert_equal 'Cardiff City FC', recs[3].name
|
|
41
|
-
## assert_equal 'Cardiff', recs[3].city.name
|
|
42
|
-
assert_equal 'Wales', recs[3].country.name
|
|
43
|
-
end
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
def test_eng_ii
|
|
47
|
-
headers = {
|
|
48
|
-
date: 'Date',
|
|
49
|
-
team1: 'HomeTeam',
|
|
50
|
-
team2: 'AwayTeam',
|
|
51
|
-
score1: 'FTHG',
|
|
52
|
-
score2: 'FTAG',
|
|
53
|
-
score1i: 'HTHG',
|
|
54
|
-
score2i: 'HTAG'
|
|
55
|
-
}
|
|
56
|
-
|
|
57
|
-
matches = SportDb::CsvMatchParser.read( "#{SportDb::Test.data_dir}/england/2017-18/E0.csv",
|
|
58
|
-
headers: headers
|
|
59
|
-
)
|
|
60
|
-
|
|
61
|
-
## pp matches[0..2]
|
|
62
|
-
|
|
63
|
-
matchlist = SportDb::Import::Matchlist.new( matches )
|
|
64
|
-
team_names = matchlist.teams
|
|
65
|
-
puts "#{team_names.size} team names:"
|
|
66
|
-
pp team_names
|
|
67
|
-
=begin
|
|
68
|
-
20 team names:
|
|
69
|
-
["Arsenal",
|
|
70
|
-
"Bournemouth",
|
|
71
|
-
"Brighton",
|
|
72
|
-
"Burnley",
|
|
73
|
-
"Chelsea",
|
|
74
|
-
"Crystal Palace",
|
|
75
|
-
"Everton",
|
|
76
|
-
"Huddersfield",
|
|
77
|
-
"Leicester",
|
|
78
|
-
"Liverpool",
|
|
79
|
-
"Man City",
|
|
80
|
-
"Man United",
|
|
81
|
-
"Newcastle",
|
|
82
|
-
"Southampton",
|
|
83
|
-
"Stoke",
|
|
84
|
-
"Swansea",
|
|
85
|
-
"Tottenham",
|
|
86
|
-
"Watford",
|
|
87
|
-
"West Brom",
|
|
88
|
-
"West Ham"]
|
|
89
|
-
=end
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
## fetch English Premiere League
|
|
93
|
-
league = LEAGUES.find!( 'ENG' )
|
|
94
|
-
|
|
95
|
-
recs = TEAMS.find_by!( name: team_names,
|
|
96
|
-
league: league )
|
|
97
|
-
|
|
98
|
-
assert_equal 20, recs.size
|
|
99
|
-
|
|
100
|
-
assert_equal 'Arsenal FC', recs[0].name
|
|
101
|
-
## assert_equal '?', recs[0].city.name
|
|
102
|
-
assert_equal 'England', recs[0].country.name
|
|
103
|
-
|
|
104
|
-
assert_equal 'AFC Bournemouth', recs[1].name
|
|
105
|
-
## assert_equal '?', recs[1].city.name
|
|
106
|
-
assert_equal 'England', recs[1].country.name
|
|
107
|
-
end
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
def test_at
|
|
111
|
-
headers = {
|
|
112
|
-
## season: 'Season', ## check if season required / needed???
|
|
113
|
-
date: 'Date',
|
|
114
|
-
team1: 'Home',
|
|
115
|
-
team2: 'Away',
|
|
116
|
-
score1: 'HG',
|
|
117
|
-
score2: 'AG',
|
|
118
|
-
}
|
|
119
|
-
|
|
120
|
-
matches = SportDb::CsvMatchParser.read( "#{SportDb::Test.data_dir}/austria/AUT.csv",
|
|
121
|
-
headers: headers,
|
|
122
|
-
filters: { 'Season' => '2016/2017' }
|
|
123
|
-
)
|
|
124
|
-
|
|
125
|
-
## pp matches[0..2]
|
|
126
|
-
|
|
127
|
-
matchlist = SportDb::Import::Matchlist.new( matches )
|
|
128
|
-
team_names = matchlist.teams
|
|
129
|
-
puts "#{team_names.size} team names:"
|
|
130
|
-
pp team_names
|
|
131
|
-
=begin
|
|
132
|
-
10 team names:
|
|
133
|
-
["AC Wolfsberger",
|
|
134
|
-
"Admira",
|
|
135
|
-
"Altach",
|
|
136
|
-
"Austria Vienna",
|
|
137
|
-
"Mattersburg",
|
|
138
|
-
"Rapid Vienna",
|
|
139
|
-
"Ried",
|
|
140
|
-
"Salzburg",
|
|
141
|
-
"St. Polten",
|
|
142
|
-
"Sturm Graz"]
|
|
143
|
-
=end
|
|
144
|
-
|
|
145
|
-
## fetch Österr. Bundesliga
|
|
146
|
-
league = LEAGUES.find!( 'AT' )
|
|
147
|
-
|
|
148
|
-
recs = TEAMS.find_by!( name: team_names,
|
|
149
|
-
league: league )
|
|
150
|
-
|
|
151
|
-
assert_equal 10, recs.size
|
|
152
|
-
|
|
153
|
-
assert_equal 'Wolfsberger AC', recs[0].name
|
|
154
|
-
## assert_equal '?', recs[0].city.name
|
|
155
|
-
assert_equal 'Austria', recs[0].country.name
|
|
156
|
-
|
|
157
|
-
assert_equal 'FC Admira Wacker Mödling', recs[1].name
|
|
158
|
-
## assert_equal '?', recs[1].city.name
|
|
159
|
-
assert_equal 'Austria', recs[1].country.name
|
|
160
|
-
end
|
|
161
|
-
|
|
162
|
-
end # class TestClub
|
|
1
|
+
###
|
|
2
|
+
# to run use
|
|
3
|
+
# ruby -I ./lib -I ./test test/test_club.rb
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
require 'helper'
|
|
7
|
+
|
|
8
|
+
class TestClub < MiniTest::Test
|
|
9
|
+
|
|
10
|
+
def test_eng_i
|
|
11
|
+
## todo/fix:
|
|
12
|
+
## add guest1_country_id (optional) to league (e.g. wales for english premier leaguage)
|
|
13
|
+
## (e.g. canada for mls)
|
|
14
|
+
## (e.g. liechtenstein for swiss superleague) etc.
|
|
15
|
+
## why? why not?
|
|
16
|
+
|
|
17
|
+
## fetch English Premiere League
|
|
18
|
+
league = LEAGUES.find!( 'ENG' )
|
|
19
|
+
|
|
20
|
+
team_names = [
|
|
21
|
+
'Manchester City',
|
|
22
|
+
'Arsenal',
|
|
23
|
+
'Liverpool',
|
|
24
|
+
'Cardiff',
|
|
25
|
+
]
|
|
26
|
+
|
|
27
|
+
recs = TEAMS.find_by!( name: team_names,
|
|
28
|
+
league: league )
|
|
29
|
+
|
|
30
|
+
assert_equal 4, recs.size
|
|
31
|
+
|
|
32
|
+
assert_equal 'Manchester City FC', recs[0].name
|
|
33
|
+
## assert_equal 'Manchester', recs[0].city.name
|
|
34
|
+
assert_equal 'England', recs[0].country.name
|
|
35
|
+
|
|
36
|
+
assert_equal 'Arsenal FC', recs[1].name
|
|
37
|
+
## assert_equal 'London', recs[1].city.name
|
|
38
|
+
assert_equal 'England', recs[1].country.name
|
|
39
|
+
|
|
40
|
+
assert_equal 'Cardiff City FC', recs[3].name
|
|
41
|
+
## assert_equal 'Cardiff', recs[3].city.name
|
|
42
|
+
assert_equal 'Wales', recs[3].country.name
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
def test_eng_ii
|
|
47
|
+
headers = {
|
|
48
|
+
date: 'Date',
|
|
49
|
+
team1: 'HomeTeam',
|
|
50
|
+
team2: 'AwayTeam',
|
|
51
|
+
score1: 'FTHG',
|
|
52
|
+
score2: 'FTAG',
|
|
53
|
+
score1i: 'HTHG',
|
|
54
|
+
score2i: 'HTAG'
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
matches = SportDb::CsvMatchParser.read( "#{SportDb::Test.data_dir}/england/2017-18/E0.csv",
|
|
58
|
+
headers: headers
|
|
59
|
+
)
|
|
60
|
+
|
|
61
|
+
## pp matches[0..2]
|
|
62
|
+
|
|
63
|
+
matchlist = SportDb::Import::Matchlist.new( matches )
|
|
64
|
+
team_names = matchlist.teams
|
|
65
|
+
puts "#{team_names.size} team names:"
|
|
66
|
+
pp team_names
|
|
67
|
+
=begin
|
|
68
|
+
20 team names:
|
|
69
|
+
["Arsenal",
|
|
70
|
+
"Bournemouth",
|
|
71
|
+
"Brighton",
|
|
72
|
+
"Burnley",
|
|
73
|
+
"Chelsea",
|
|
74
|
+
"Crystal Palace",
|
|
75
|
+
"Everton",
|
|
76
|
+
"Huddersfield",
|
|
77
|
+
"Leicester",
|
|
78
|
+
"Liverpool",
|
|
79
|
+
"Man City",
|
|
80
|
+
"Man United",
|
|
81
|
+
"Newcastle",
|
|
82
|
+
"Southampton",
|
|
83
|
+
"Stoke",
|
|
84
|
+
"Swansea",
|
|
85
|
+
"Tottenham",
|
|
86
|
+
"Watford",
|
|
87
|
+
"West Brom",
|
|
88
|
+
"West Ham"]
|
|
89
|
+
=end
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
## fetch English Premiere League
|
|
93
|
+
league = LEAGUES.find!( 'ENG' )
|
|
94
|
+
|
|
95
|
+
recs = TEAMS.find_by!( name: team_names,
|
|
96
|
+
league: league )
|
|
97
|
+
|
|
98
|
+
assert_equal 20, recs.size
|
|
99
|
+
|
|
100
|
+
assert_equal 'Arsenal FC', recs[0].name
|
|
101
|
+
## assert_equal '?', recs[0].city.name
|
|
102
|
+
assert_equal 'England', recs[0].country.name
|
|
103
|
+
|
|
104
|
+
assert_equal 'AFC Bournemouth', recs[1].name
|
|
105
|
+
## assert_equal '?', recs[1].city.name
|
|
106
|
+
assert_equal 'England', recs[1].country.name
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
def test_at
|
|
111
|
+
headers = {
|
|
112
|
+
## season: 'Season', ## check if season required / needed???
|
|
113
|
+
date: 'Date',
|
|
114
|
+
team1: 'Home',
|
|
115
|
+
team2: 'Away',
|
|
116
|
+
score1: 'HG',
|
|
117
|
+
score2: 'AG',
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
matches = SportDb::CsvMatchParser.read( "#{SportDb::Test.data_dir}/austria/AUT.csv",
|
|
121
|
+
headers: headers,
|
|
122
|
+
filters: { 'Season' => '2016/2017' }
|
|
123
|
+
)
|
|
124
|
+
|
|
125
|
+
## pp matches[0..2]
|
|
126
|
+
|
|
127
|
+
matchlist = SportDb::Import::Matchlist.new( matches )
|
|
128
|
+
team_names = matchlist.teams
|
|
129
|
+
puts "#{team_names.size} team names:"
|
|
130
|
+
pp team_names
|
|
131
|
+
=begin
|
|
132
|
+
10 team names:
|
|
133
|
+
["AC Wolfsberger",
|
|
134
|
+
"Admira",
|
|
135
|
+
"Altach",
|
|
136
|
+
"Austria Vienna",
|
|
137
|
+
"Mattersburg",
|
|
138
|
+
"Rapid Vienna",
|
|
139
|
+
"Ried",
|
|
140
|
+
"Salzburg",
|
|
141
|
+
"St. Polten",
|
|
142
|
+
"Sturm Graz"]
|
|
143
|
+
=end
|
|
144
|
+
|
|
145
|
+
## fetch Österr. Bundesliga
|
|
146
|
+
league = LEAGUES.find!( 'AT' )
|
|
147
|
+
|
|
148
|
+
recs = TEAMS.find_by!( name: team_names,
|
|
149
|
+
league: league )
|
|
150
|
+
|
|
151
|
+
assert_equal 10, recs.size
|
|
152
|
+
|
|
153
|
+
assert_equal 'Wolfsberger AC', recs[0].name
|
|
154
|
+
## assert_equal '?', recs[0].city.name
|
|
155
|
+
assert_equal 'Austria', recs[0].country.name
|
|
156
|
+
|
|
157
|
+
assert_equal 'FC Admira Wacker Mödling', recs[1].name
|
|
158
|
+
## assert_equal '?', recs[1].city.name
|
|
159
|
+
assert_equal 'Austria', recs[1].country.name
|
|
160
|
+
end
|
|
161
|
+
|
|
162
|
+
end # class TestClub
|
data/test/test_import.rb
CHANGED
|
@@ -1,55 +1,55 @@
|
|
|
1
|
-
# encoding: utf-8
|
|
2
|
-
|
|
3
|
-
###
|
|
4
|
-
# to run use
|
|
5
|
-
# ruby -I ./lib -I ./test test/test_import.rb
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
require 'helper'
|
|
9
|
-
|
|
10
|
-
class TestImport < MiniTest::Test
|
|
11
|
-
|
|
12
|
-
def setup
|
|
13
|
-
SportDb.connect( adapter: 'sqlite3',
|
|
14
|
-
database: ':memory:' )
|
|
15
|
-
SportDb.create_all ## build schema
|
|
16
|
-
|
|
17
|
-
## turn on logging to console
|
|
18
|
-
## ActiveRecord::Base.logger = Logger.new(STDOUT)
|
|
19
|
-
end
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
def test_eng
|
|
23
|
-
## fix/todo:
|
|
24
|
-
## use Event.read_csv - why? why not?
|
|
25
|
-
## Event.parse_csv - why? why not?
|
|
26
|
-
|
|
27
|
-
headers = {
|
|
28
|
-
date: 'Date',
|
|
29
|
-
team1: 'HomeTeam',
|
|
30
|
-
team2: 'AwayTeam',
|
|
31
|
-
score1: 'FTHG',
|
|
32
|
-
score2: 'FTAG',
|
|
33
|
-
score1i: 'HTHG',
|
|
34
|
-
score2i: 'HTAG'
|
|
35
|
-
}
|
|
36
|
-
|
|
37
|
-
event_rec = SportDb::CsvEventImporter.read( "#{SportDb::Test.data_dir}/england/2017-18/E0.csv",
|
|
38
|
-
headers: headers,
|
|
39
|
-
league: 'ENG', ## fetch English Premiere League
|
|
40
|
-
season: '2017/18'
|
|
41
|
-
)
|
|
42
|
-
|
|
43
|
-
assert_equal 20, event_rec.teams.count
|
|
44
|
-
assert_equal 380, event_rec.matches.count
|
|
45
|
-
|
|
46
|
-
rec = event_rec.matches.first
|
|
47
|
-
assert_equal 'Arsenal FC', rec.team1.name
|
|
48
|
-
assert_equal 'Leicester City FC', rec.team2.name
|
|
49
|
-
assert_equal 4, rec.score1
|
|
50
|
-
assert_equal 3, rec.score2
|
|
51
|
-
assert_equal 2, rec.score1i
|
|
52
|
-
assert_equal 2, rec.score2i
|
|
53
|
-
end
|
|
54
|
-
|
|
55
|
-
end # class TestImport
|
|
1
|
+
# encoding: utf-8
|
|
2
|
+
|
|
3
|
+
###
|
|
4
|
+
# to run use
|
|
5
|
+
# ruby -I ./lib -I ./test test/test_import.rb
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
require 'helper'
|
|
9
|
+
|
|
10
|
+
class TestImport < MiniTest::Test
|
|
11
|
+
|
|
12
|
+
def setup
|
|
13
|
+
SportDb.connect( adapter: 'sqlite3',
|
|
14
|
+
database: ':memory:' )
|
|
15
|
+
SportDb.create_all ## build schema
|
|
16
|
+
|
|
17
|
+
## turn on logging to console
|
|
18
|
+
## ActiveRecord::Base.logger = Logger.new(STDOUT)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
def test_eng
|
|
23
|
+
## fix/todo:
|
|
24
|
+
## use Event.read_csv - why? why not?
|
|
25
|
+
## Event.parse_csv - why? why not?
|
|
26
|
+
|
|
27
|
+
headers = {
|
|
28
|
+
date: 'Date',
|
|
29
|
+
team1: 'HomeTeam',
|
|
30
|
+
team2: 'AwayTeam',
|
|
31
|
+
score1: 'FTHG',
|
|
32
|
+
score2: 'FTAG',
|
|
33
|
+
score1i: 'HTHG',
|
|
34
|
+
score2i: 'HTAG'
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
event_rec = SportDb::CsvEventImporter.read( "#{SportDb::Test.data_dir}/england/2017-18/E0.csv",
|
|
38
|
+
headers: headers,
|
|
39
|
+
league: 'ENG', ## fetch English Premiere League
|
|
40
|
+
season: '2017/18'
|
|
41
|
+
)
|
|
42
|
+
|
|
43
|
+
assert_equal 20, event_rec.teams.count
|
|
44
|
+
assert_equal 380, event_rec.matches.count
|
|
45
|
+
|
|
46
|
+
rec = event_rec.matches.first
|
|
47
|
+
assert_equal 'Arsenal FC', rec.team1.name
|
|
48
|
+
assert_equal 'Leicester City FC', rec.team2.name
|
|
49
|
+
assert_equal 4, rec.score1
|
|
50
|
+
assert_equal 3, rec.score2
|
|
51
|
+
assert_equal 2, rec.score1i
|
|
52
|
+
assert_equal 2, rec.score2i
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
end # class TestImport
|
data/test/test_version.rb
CHANGED
|
@@ -1,22 +1,22 @@
|
|
|
1
|
-
# encoding: utf-8
|
|
2
|
-
|
|
3
|
-
###
|
|
4
|
-
# to run use
|
|
5
|
-
# ruby -I ./lib -I ./test test/test_version.rb
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
require 'helper'
|
|
9
|
-
|
|
10
|
-
class TestVersion < MiniTest::Test
|
|
11
|
-
|
|
12
|
-
def test_version
|
|
13
|
-
pp SportDb::Module::Importers::VERSION
|
|
14
|
-
pp SportDb::Module::Importers.banner
|
|
15
|
-
pp SportDb::Module::Importers.root
|
|
16
|
-
|
|
17
|
-
pp SportDb::Module.constants
|
|
18
|
-
|
|
19
|
-
assert true
|
|
20
|
-
end
|
|
21
|
-
|
|
22
|
-
end # class TestVersion
|
|
1
|
+
# encoding: utf-8
|
|
2
|
+
|
|
3
|
+
###
|
|
4
|
+
# to run use
|
|
5
|
+
# ruby -I ./lib -I ./test test/test_version.rb
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
require 'helper'
|
|
9
|
+
|
|
10
|
+
class TestVersion < MiniTest::Test
|
|
11
|
+
|
|
12
|
+
def test_version
|
|
13
|
+
pp SportDb::Module::Importers::VERSION
|
|
14
|
+
pp SportDb::Module::Importers.banner
|
|
15
|
+
pp SportDb::Module::Importers.root
|
|
16
|
+
|
|
17
|
+
pp SportDb::Module.constants
|
|
18
|
+
|
|
19
|
+
assert true
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
end # class TestVersion
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: sportdb-importers
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Gerald Bauer
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2020-
|
|
11
|
+
date: 2020-11-14 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: sportdb-sync
|
|
@@ -28,30 +28,36 @@ dependencies:
|
|
|
28
28
|
name: rdoc
|
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
|
30
30
|
requirements:
|
|
31
|
-
- - "
|
|
31
|
+
- - ">="
|
|
32
32
|
- !ruby/object:Gem::Version
|
|
33
33
|
version: '4.0'
|
|
34
|
+
- - "<"
|
|
35
|
+
- !ruby/object:Gem::Version
|
|
36
|
+
version: '7'
|
|
34
37
|
type: :development
|
|
35
38
|
prerelease: false
|
|
36
39
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
40
|
requirements:
|
|
38
|
-
- - "
|
|
41
|
+
- - ">="
|
|
39
42
|
- !ruby/object:Gem::Version
|
|
40
43
|
version: '4.0'
|
|
44
|
+
- - "<"
|
|
45
|
+
- !ruby/object:Gem::Version
|
|
46
|
+
version: '7'
|
|
41
47
|
- !ruby/object:Gem::Dependency
|
|
42
48
|
name: hoe
|
|
43
49
|
requirement: !ruby/object:Gem::Requirement
|
|
44
50
|
requirements:
|
|
45
51
|
- - "~>"
|
|
46
52
|
- !ruby/object:Gem::Version
|
|
47
|
-
version: '3.
|
|
53
|
+
version: '3.22'
|
|
48
54
|
type: :development
|
|
49
55
|
prerelease: false
|
|
50
56
|
version_requirements: !ruby/object:Gem::Requirement
|
|
51
57
|
requirements:
|
|
52
58
|
- - "~>"
|
|
53
59
|
- !ruby/object:Gem::Version
|
|
54
|
-
version: '3.
|
|
60
|
+
version: '3.22'
|
|
55
61
|
description: sportdb-importers - tools 'n' scripts for importing sports (football)
|
|
56
62
|
data in alternate (text) formats incl. comma-separated values (csv) format
|
|
57
63
|
email: opensport@googlegroups.com
|