limeta 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/Gemfile +3 -0
- data/README.md +189 -0
- data/Rakefile +1 -0
- data/bin/limeta +8 -0
- data/lib/limeta.rb +2 -0
- data/lib/limeta/cli.rb +126 -0
- data/lib/limeta/connection.rb +13 -0
- data/lib/limeta/table.rb +128 -0
- data/lib/limeta/version.rb +8 -0
- data/limeta.gemspec +40 -0
- metadata +211 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 0a0e62434b766a62847359621c2ffe3d6692f2943cf21da8e82e364a8b0640d1
|
4
|
+
data.tar.gz: 12d07101127835f532f58497b3d03364be1741bdb9dfd43992390843423c2f7e
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: dd4c01715f7035ad6b4d15522cce3ad5b1a4007b77f2ff920fc1ae24d52bb3dd906098cf5e6f202bf26a76d077b3ced68cf7f722f7e68383bc09f9739c118ca4
|
7
|
+
data.tar.gz: e3b5d3638a48be0ac04046d95be6e12ea1243e2414ac8cd8bf68f1a88cf0348ed3f481183608705343e3939ec635553e6eebfb4e01dee41964e5beb1c5fe2026
|
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,189 @@
|
|
1
|
+
![Gem](https://img.shields.io/gem/v/limeta.svg)
|
2
|
+
[![License](https://img.shields.io/github/license/eonu/limeta.svg)](https://github.com/eonu/limeta/blob/master/LICENSE)
|
3
|
+
|
4
|
+
# Limeta
|
5
|
+
|
6
|
+
Limeta (**Li**chess **meta**data) is a conversion tool for extracting chess game metadata from Lichess' public PGN files and exporting it to database tables.
|
7
|
+
|
8
|
+
## What is a PGN file?
|
9
|
+
|
10
|
+
PGN (**P**ortable **G**ame **N**otation) files are plain-text files that provide records of chess games along with their metadata.
|
11
|
+
|
12
|
+
Although PGN files generally have the same structure, there are many variations which store different metadata fields.
|
13
|
+
|
14
|
+
### Lichess PGN format
|
15
|
+
|
16
|
+
The extensive [game database provided by Lichess](https://database.lichess.org/) (over 740 million games as of July 2019) uses the following format for each game of within their PGN files, each storing the data of millions of chess games.
|
17
|
+
|
18
|
+
This format is essentially the same as standard PGN formats, but with a few differences - particularly with the record moves section and some of the metadata fields.
|
19
|
+
|
20
|
+
```
|
21
|
+
[Event "Rated Bullet tournament https://lichess.org/tournament/yc1WW2Ox"]
|
22
|
+
[Site "https://lichess.org/PpwPOZMq"]
|
23
|
+
[White "Abbot"]
|
24
|
+
[Black "Costello"]
|
25
|
+
[Result "0-1"]
|
26
|
+
[UTCDate "2017.04.01"]
|
27
|
+
[UTCTime "11:32:01"]
|
28
|
+
[WhiteElo "2100"]
|
29
|
+
[BlackElo "2000"]
|
30
|
+
[WhiteRatingDiff "-4"]
|
31
|
+
[BlackRatingDiff "+1"]
|
32
|
+
[WhiteTitle "FM"]
|
33
|
+
[BlackTitle "GM"]
|
34
|
+
[ECO "B30"]
|
35
|
+
[Opening "Sicilian Defense: Old Sicilian"]
|
36
|
+
[TimeControl "300+0"]
|
37
|
+
[Termination "Time forfeit"]
|
38
|
+
|
39
|
+
1. e4 { [%eval 0.17] [%clk 0:00:30] } 1... c5 { [%eval 0.19] [%clk 0:00:30] }
|
40
|
+
2. Nf3 { [%eval 0.25] [%clk 0:00:29] } 2... Nc6 { [%eval 0.33] [%clk 0:00:30] }
|
41
|
+
3. Bc4 { [%eval -0.13] [%clk 0:00:28] } 3... e6 { [%eval -0.04] [%clk 0:00:30] }
|
42
|
+
4. c3 { [%eval -0.4] [%clk 0:00:27] } 4... b5? { [%eval 1.18] [%clk 0:00:30] }
|
43
|
+
5. Bb3?! { [%eval 0.21] [%clk 0:00:26] } 5... c4 { [%eval 0.32] [%clk 0:00:29] }
|
44
|
+
6. Bc2 { [%eval 0.2] [%clk 0:00:25] } 6... a5 { [%eval 0.6] [%clk 0:00:29] }
|
45
|
+
7. d4 { [%eval 0.29] [%clk 0:00:23] } 7... cxd3 { [%eval 0.6] [%clk 0:00:27] }
|
46
|
+
8. Qxd3 { [%eval 0.12] [%clk 0:00:22] } 8... Nf6 { [%eval 0.52] [%clk 0:00:26] }
|
47
|
+
9. e5 { [%eval 0.39] [%clk 0:00:21] } 9... Nd5 { [%eval 0.45] [%clk 0:00:25] }
|
48
|
+
10. Bg5?! { [%eval -0.44] [%clk 0:00:18] } 10... Qc7 { [%eval -0.12] [%clk 0:00:23] }
|
49
|
+
11. Nbd2?? { [%eval -3.15] [%clk 0:00:14] } 11... h6 { [%eval -2.99] [%clk 0:00:23] }
|
50
|
+
12. Bh4 { [%eval -3.0] [%clk 0:00:11] } 12... Ba6? { [%eval -0.12] [%clk 0:00:23] }
|
51
|
+
13. b3?? { [%eval -4.14] [%clk 0:00:02] } 13... Nf4? { [%eval -2.73] [%clk 0:00:21] } 0-1
|
52
|
+
```
|
53
|
+
|
54
|
+
The data is divided into **two** sections:
|
55
|
+
|
56
|
+
#### Metadata
|
57
|
+
|
58
|
+
This section consists of data about the chess game itself along with information about the players of the game.
|
59
|
+
|
60
|
+
```
|
61
|
+
[Event "Rated Bullet tournament https://lichess.org/tournament/yc1WW2Ox"]
|
62
|
+
[Site "https://lichess.org/PpwPOZMq"]
|
63
|
+
[White "Abbot"]
|
64
|
+
[Black "Costello"]
|
65
|
+
[Result "0-1"]
|
66
|
+
[UTCDate "2017.04.01"]
|
67
|
+
[UTCTime "11:32:01"]
|
68
|
+
[WhiteElo "2100"]
|
69
|
+
[BlackElo "2000"]
|
70
|
+
[WhiteRatingDiff "-4"]
|
71
|
+
[BlackRatingDiff "+1"]
|
72
|
+
[WhiteTitle "FM"]
|
73
|
+
[BlackTitle "GM"]
|
74
|
+
[ECO "B30"]
|
75
|
+
[Opening "Sicilian Defense: Old Sicilian"]
|
76
|
+
[TimeControl "300+0"]
|
77
|
+
[Termination "Time forfeit"]
|
78
|
+
```
|
79
|
+
|
80
|
+
Limeta extracts these individual key-value pairs and stores them in a record for each game in the PGN file, where the fields of the table represent the keys above.
|
81
|
+
|
82
|
+
#### Game record
|
83
|
+
|
84
|
+
The game record is an ordered sequence of the moves played in an individual chess game. These moves are recorded in standard algebraic notation (SAN). In the case of Lichess PGN files, chess engine evaluations are also sometimes included for each move (if enabled for that game).
|
85
|
+
|
86
|
+
```
|
87
|
+
1. e4 { [%eval 0.17] [%clk 0:00:30] } 1... c5 { [%eval 0.19] [%clk 0:00:30] }
|
88
|
+
2. Nf3 { [%eval 0.25] [%clk 0:00:29] } 2... Nc6 { [%eval 0.33] [%clk 0:00:30] }
|
89
|
+
3. Bc4 { [%eval -0.13] [%clk 0:00:28] } 3... e6 { [%eval -0.04] [%clk 0:00:30] }
|
90
|
+
4. c3 { [%eval -0.4] [%clk 0:00:27] } 4... b5? { [%eval 1.18] [%clk 0:00:30] }
|
91
|
+
5. Bb3?! { [%eval 0.21] [%clk 0:00:26] } 5... c4 { [%eval 0.32] [%clk 0:00:29] }
|
92
|
+
6. Bc2 { [%eval 0.2] [%clk 0:00:25] } 6... a5 { [%eval 0.6] [%clk 0:00:29] }
|
93
|
+
7. d4 { [%eval 0.29] [%clk 0:00:23] } 7... cxd3 { [%eval 0.6] [%clk 0:00:27] }
|
94
|
+
8. Qxd3 { [%eval 0.12] [%clk 0:00:22] } 8... Nf6 { [%eval 0.52] [%clk 0:00:26] }
|
95
|
+
9. e5 { [%eval 0.39] [%clk 0:00:21] } 9... Nd5 { [%eval 0.45] [%clk 0:00:25] }
|
96
|
+
10. Bg5?! { [%eval -0.44] [%clk 0:00:18] } 10... Qc7 { [%eval -0.12] [%clk 0:00:23] }
|
97
|
+
11. Nbd2?? { [%eval -3.15] [%clk 0:00:14] } 11... h6 { [%eval -2.99] [%clk 0:00:23] }
|
98
|
+
12. Bh4 { [%eval -3.0] [%clk 0:00:11] } 12... Ba6? { [%eval -0.12] [%clk 0:00:23] }
|
99
|
+
13. b3?? { [%eval -4.14] [%clk 0:00:02] } 13... Nf4? { [%eval -2.73] [%clk 0:00:21] } 0-1
|
100
|
+
```
|
101
|
+
|
102
|
+
As Limeta is only concerned with the metadata of chess games, this section is discarded during the parsing process.
|
103
|
+
|
104
|
+
## Installation
|
105
|
+
|
106
|
+
To install the CLI:
|
107
|
+
|
108
|
+
```bash
|
109
|
+
$ gem install limeta
|
110
|
+
```
|
111
|
+
|
112
|
+
## Usage
|
113
|
+
|
114
|
+
Once you have installed the CLI, you can inspect it using the command:
|
115
|
+
|
116
|
+
```bash
|
117
|
+
$ limeta -h
|
118
|
+
Commands:
|
119
|
+
limeta [FILES] # Export PGN metadata to a database
|
120
|
+
limeta --version, -v # Display installed Limeta version
|
121
|
+
|
122
|
+
Options:
|
123
|
+
-a, --adapter=ADAPTER # Database adapter
|
124
|
+
# Possible values: psql, sqlite3
|
125
|
+
-c, [--conn=CONN] # Connection string
|
126
|
+
-t, --table=TABLE # Export table name
|
127
|
+
```
|
128
|
+
|
129
|
+
Limeta supports two database adapters: `sqlite3` and `psql`.
|
130
|
+
|
131
|
+
### Options
|
132
|
+
|
133
|
+
- `--adapter, -a`: **Database adapter** (Required)
|
134
|
+
The database adapter to use for connecting to the database.<br/>_Must be either `sqlite3` or `psql`_.
|
135
|
+
- `--table, -t`, **Table name** (Required)
|
136
|
+
The name of the database table to export the PGN metadata to.
|
137
|
+
- `--conn, -c`: **Connection string** (Not required)
|
138
|
+
The connection string for connecting to the database through the specified adapter. If not provided as an option argument, the user will be prompted to enter the information to form a connection string.
|
139
|
+
|
140
|
+
If using the `sqlite3` adapter, the connection string is simply a path to the desired database file (with extension `.sqlite3`, `.sqlite`, `.db` or `.sql`).
|
141
|
+
|
142
|
+
> **Example**: `/Users/eonu/development/chess/data/lichess.sqlite3`
|
143
|
+
|
144
|
+
If using the `psql` adapter, the connection string is in the form of `postgresql://[user[:password]@][netloc][:port][/dbname]`. Read more about PostgreSQL connection strings [here](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING).
|
145
|
+
|
146
|
+
> **Examples**:
|
147
|
+
> - `postgresql://eonu:munchy7@localhost:5432/chess`
|
148
|
+
> - `postgresql://eonu@46.101.90.215:5432/chess`
|
149
|
+
|
150
|
+
### Examples
|
151
|
+
|
152
|
+
```bash
|
153
|
+
# Retrieves all PGN files in the current directory
|
154
|
+
$ limeta *.pgn -a sqlite3 -t games -c /Users/eonu/development/chess/data/lichess.sqlite3
|
155
|
+
|
156
|
+
# Recursively searches for PGN files in current directory and subdirectories
|
157
|
+
$ limeta . -a psql -t games -c postgresql://eonu@46.101.90.215:5432/chess
|
158
|
+
|
159
|
+
# Populates the table using the 2018-02.pgn file after prompting the user for a PostgreSQL connection string
|
160
|
+
$ limeta 2018-02.pgn -a psql -t games
|
161
|
+
```
|
162
|
+
|
163
|
+
# Contributors
|
164
|
+
|
165
|
+
All contributions to this repository are greatly appreciated. Contribution guidelines can be found [here](/CONTRIBUTING.md).
|
166
|
+
|
167
|
+
<table>
|
168
|
+
<thead>
|
169
|
+
<tr>
|
170
|
+
<th align="center">
|
171
|
+
<a href="https://github.com/eonu">
|
172
|
+
<img src="https://avatars0.githubusercontent.com/u/24795571?s=460&v=4" alt="Edwin Onuonga" width="60px">
|
173
|
+
<br/><sub><b>Edwin Onuonga</b></sub>
|
174
|
+
</a>
|
175
|
+
<br/>
|
176
|
+
<a href="mailto:ed@eonu.net">✉️</a>
|
177
|
+
<a href="https://eonu.net">🌍</a>
|
178
|
+
</th>
|
179
|
+
<!-- Add more <th></th> blocks for more contributors -->
|
180
|
+
</tr>
|
181
|
+
</thead>
|
182
|
+
</table>
|
183
|
+
|
184
|
+
---
|
185
|
+
|
186
|
+
<p align="center">
|
187
|
+
© 2019, Edwin Onuonga - Released under the <a href="https://opensource.org/licenses/MIT">MIT</a> License.<br/>
|
188
|
+
<em>Authored and maintained by Edwin Onuonga.</em>
|
189
|
+
</p>
|
data/Rakefile
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
require "bundler/gem_tasks"
|
data/bin/limeta
ADDED
@@ -0,0 +1,8 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
lib = File.expand_path("../../lib", __FILE__)
|
3
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
+
require 'limeta'
|
5
|
+
|
6
|
+
ARGV.unshift(Limeta::CLI.default_task) unless Limeta::CLI.all_tasks.has_key?(ARGV[0]) || Limeta::CLI.instance_variable_get(:@map).has_key?(ARGV[0])
|
7
|
+
|
8
|
+
Limeta::CLI.start(ARGV)
|
data/lib/limeta.rb
ADDED
data/lib/limeta/cli.rb
ADDED
@@ -0,0 +1,126 @@
|
|
1
|
+
require 'thor'
|
2
|
+
require 'fileutils'
|
3
|
+
require 'pg/url'
|
4
|
+
require 'string/builder'
|
5
|
+
require 'tty/prompt'
|
6
|
+
require 'limeta/connection'
|
7
|
+
require 'limeta/table'
|
8
|
+
|
9
|
+
module Limeta
|
10
|
+
class CLI < Thor
|
11
|
+
using String::Builder
|
12
|
+
|
13
|
+
default_task :convert
|
14
|
+
|
15
|
+
class_option :adapter, type: :string, aliases: '-a', desc: 'Database adapter', enum: %w[psql sqlite3], required: true
|
16
|
+
class_option :conn, type: :string, aliases: '-c', desc: 'Connection string', required: false
|
17
|
+
class_option :table, type: :string, aliases: '-t', desc: 'Export table name', required: true
|
18
|
+
|
19
|
+
disable_required_check! :help, :version
|
20
|
+
|
21
|
+
desc "[FILES]", "Export PGN metadata to a database"
|
22
|
+
def convert(*files)
|
23
|
+
files = expand files
|
24
|
+
raise ArgumentError.new("No PGN files provided") if files.empty?
|
25
|
+
|
26
|
+
prompt = TTY::Prompt.new
|
27
|
+
|
28
|
+
conn = case options[:adapter]
|
29
|
+
when 'psql'
|
30
|
+
options[:conn] || PG::URL.prompt
|
31
|
+
when 'sqlite3'
|
32
|
+
options[:conn] || String.build do |s|
|
33
|
+
path = File.expand_path prompt.ask("Path to database file:")
|
34
|
+
extensions = %w[.db .sql .sqlite .sqlite3]
|
35
|
+
if File.file? path
|
36
|
+
if extensions.include? File.extname(path)
|
37
|
+
s << path
|
38
|
+
else
|
39
|
+
raise "Expected #{File.basename path} extension to be one of #{extensions.inspect}"
|
40
|
+
end
|
41
|
+
elsif File.directory? path
|
42
|
+
puts
|
43
|
+
if prompt.yes? "Create a database in \e[1m#{path}\e[0m?"
|
44
|
+
db = nil
|
45
|
+
pass = false
|
46
|
+
until pass
|
47
|
+
name = prompt.ask "Database name:"
|
48
|
+
db = File.join path, "#{name.sub(/\.(sqlite3|sqlite|sql|db|)/, '')}.sqlite3"
|
49
|
+
puts
|
50
|
+
pass = prompt.yes? "Is \e[1m#{db}\e[0m correct?"
|
51
|
+
end
|
52
|
+
puts
|
53
|
+
|
54
|
+
if File.exist? db
|
55
|
+
puts "\e[31mWARNING\e[0m: Database \e[1m#{db}\e[0m already exists."
|
56
|
+
return unless prompt.yes? "Continue anyway?"
|
57
|
+
else
|
58
|
+
FileUtils.touch db
|
59
|
+
puts "Database #{db} created"
|
60
|
+
end
|
61
|
+
|
62
|
+
s << db
|
63
|
+
else
|
64
|
+
return
|
65
|
+
end
|
66
|
+
else
|
67
|
+
if File.directory? File.dirname(path)
|
68
|
+
if extensions.include? File.extname(path)
|
69
|
+
if prompt.yes? "Create database \e[1m#{path}\e[0m?"
|
70
|
+
FileUtils.touch path
|
71
|
+
puts "Database \e[1m#{path}\e[0m created"
|
72
|
+
else
|
73
|
+
return
|
74
|
+
end
|
75
|
+
else
|
76
|
+
raise "Expected #{File.basename path} extension to be one of #{extensions.inspect}"
|
77
|
+
end
|
78
|
+
else
|
79
|
+
raise "Invalid path #{path}"
|
80
|
+
end
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
conn = case options[:adapter]
|
86
|
+
when 'psql' then conn.start_with?("postgres://") ? conn : "postgres://#{conn}"
|
87
|
+
when 'sqlite3' then conn
|
88
|
+
end
|
89
|
+
|
90
|
+
db = Limeta::Connection.establish conn, adapter: options[:adapter]
|
91
|
+
table = Limeta::Table.new options[:table], database: db
|
92
|
+
table.populate! files
|
93
|
+
end
|
94
|
+
|
95
|
+
map %w[--version -v] => :version
|
96
|
+
desc '--version, -v', 'Display installed Limeta version'
|
97
|
+
def version() puts Limeta::VERSION end
|
98
|
+
|
99
|
+
no_tasks do
|
100
|
+
def expand(files)
|
101
|
+
files.map{|f| File.file?(f) ? Dir[f] : Dir["#{f}/**/*.pgn"]}.flatten
|
102
|
+
end
|
103
|
+
end
|
104
|
+
|
105
|
+
class << self
|
106
|
+
def help(shell, subcommand = false)
|
107
|
+
list = printable_commands(true, subcommand)
|
108
|
+
Thor::Util.thor_classes_in(self).each do |klass|
|
109
|
+
list += klass.printable_commands(false)
|
110
|
+
end
|
111
|
+
|
112
|
+
list.reject! {|l| l[0].split[1] == 'help'}
|
113
|
+
|
114
|
+
if defined?(@package_name) && @package_name
|
115
|
+
shell.say "#{@package_name} commands:"
|
116
|
+
else
|
117
|
+
shell.say "Commands:"
|
118
|
+
end
|
119
|
+
|
120
|
+
shell.print_table(list, indent: 2, truncate: true)
|
121
|
+
shell.say
|
122
|
+
class_options_help(shell)
|
123
|
+
end
|
124
|
+
end
|
125
|
+
end
|
126
|
+
end
|
data/lib/limeta/table.rb
ADDED
@@ -0,0 +1,128 @@
|
|
1
|
+
require 'tty/prompt'
|
2
|
+
require 'tty/progressbar'
|
3
|
+
require 'tty/progressbar/multi'
|
4
|
+
require 'string/builder'
|
5
|
+
require 'benchmark'
|
6
|
+
|
7
|
+
module Limeta
|
8
|
+
class Table
|
9
|
+
using String::Builder
|
10
|
+
|
11
|
+
FIELDS = {
|
12
|
+
Event: {type: :string},
|
13
|
+
Site: {type: :string, size: 50},
|
14
|
+
White: {type: :string, size: 30},
|
15
|
+
Black: {type: :string, size: 30},
|
16
|
+
Result: {type: :string, size: 10},
|
17
|
+
UTCDate: {type: :date, size: 15},
|
18
|
+
UTCTime: {type: :time, size: 8},
|
19
|
+
WhiteElo: {type: :integer},
|
20
|
+
BlackElo: {type: :integer},
|
21
|
+
WhiteRatingDiff: {type: :integer},
|
22
|
+
BlackRatingDiff: {type: :integer},
|
23
|
+
WhiteTitle: {type: :string, size: 5},
|
24
|
+
BlackTitle: {type: :string, size: 5},
|
25
|
+
ECO: {type: :string, size: 3},
|
26
|
+
Opening: {type: :string},
|
27
|
+
TimeControl: {type: :string, size: 15},
|
28
|
+
Termination: {type: :string, size: 20},
|
29
|
+
Variant: {type: :string}
|
30
|
+
}
|
31
|
+
|
32
|
+
def initialize(name, database:)
|
33
|
+
@name = name
|
34
|
+
@database = database
|
35
|
+
prepare
|
36
|
+
end
|
37
|
+
|
38
|
+
def populate!(files)
|
39
|
+
progress = TTY::ProgressBar::Multi.new("\e[1mAll files\e[0m: [:bar] :percent", incomplete: "\e[90m=\e[0m", complete: "\e[32;1m=\e[0m")
|
40
|
+
bars = {}
|
41
|
+
|
42
|
+
files.each do |file|
|
43
|
+
bars[file] = progress.register("#{file} [:bar] :percent (of :total bytes)", total: File.size(file), complete: "\e[32;2m=\e[0m")
|
44
|
+
end
|
45
|
+
|
46
|
+
puts
|
47
|
+
progress.start
|
48
|
+
|
49
|
+
bm = Benchmark.measure do
|
50
|
+
files.each do |file|
|
51
|
+
record = {}
|
52
|
+
prev = ''
|
53
|
+
|
54
|
+
File.open(file, 'r').each do |line|
|
55
|
+
if prev =~ /\A\[.*\Z/ && line =~ /\A[\n\r].*\Z/
|
56
|
+
@database[@name.to_sym].insert(**record)
|
57
|
+
record = {}
|
58
|
+
prev = line
|
59
|
+
bars[file].advance(line.bytesize) # Increment
|
60
|
+
next
|
61
|
+
end
|
62
|
+
if line =~ /\A([\n\r]|[0-9]).*\Z/
|
63
|
+
prev = line
|
64
|
+
bars[file].advance(line.bytesize) # Increment
|
65
|
+
next
|
66
|
+
end
|
67
|
+
subbed = line.gsub(%r{[\r\n\[\]\"]},'')
|
68
|
+
field, value = subbed.split(' ', 2)
|
69
|
+
field = field.to_sym
|
70
|
+
if FIELDS[field]
|
71
|
+
record[field] = case FIELDS[field][:type]
|
72
|
+
when :string then value
|
73
|
+
when :integer then value.to_i
|
74
|
+
when :date then Date.parse(value)
|
75
|
+
when :time then value
|
76
|
+
end
|
77
|
+
end
|
78
|
+
prev = line
|
79
|
+
bars[file].advance(line.bytesize) # Increment
|
80
|
+
end
|
81
|
+
end
|
82
|
+
end
|
83
|
+
|
84
|
+
bm_out = String.build("Finished in ") do |s|
|
85
|
+
s << "#{bm.utime.round(4)}s (cpu), "
|
86
|
+
s << "#{bm.stime.round(4)}s (system), "
|
87
|
+
s << "#{bm.total.round(4)}s (total), "
|
88
|
+
s << "#{bm.real.round(4)}s (real)"
|
89
|
+
end
|
90
|
+
|
91
|
+
puts
|
92
|
+
puts bm_out
|
93
|
+
|
94
|
+
@database.disconnect
|
95
|
+
end
|
96
|
+
|
97
|
+
def prepare
|
98
|
+
prompt = TTY::Prompt.new
|
99
|
+
if exist?
|
100
|
+
if prompt.yes? "Overwrite table '\e[1m#{@name}\e[0m'?"
|
101
|
+
if prompt.yes? "Are you sure? This action cannot be reverted."
|
102
|
+
create force: true
|
103
|
+
end
|
104
|
+
end
|
105
|
+
else
|
106
|
+
create
|
107
|
+
end
|
108
|
+
end
|
109
|
+
|
110
|
+
def exist?
|
111
|
+
@database.table_exists? @name
|
112
|
+
end
|
113
|
+
|
114
|
+
def create(force: false)
|
115
|
+
@database.send((force ? :create_table! : :create_table), @name) do
|
116
|
+
primary_key :id
|
117
|
+
FIELDS.each do |name, options|
|
118
|
+
case options[:type]
|
119
|
+
when :string then String name, size: (options[:size] || 255)
|
120
|
+
when :integer then Integer name
|
121
|
+
when :date then Date name
|
122
|
+
when :time then Time name, only_time: true
|
123
|
+
end
|
124
|
+
end
|
125
|
+
end
|
126
|
+
end
|
127
|
+
end
|
128
|
+
end
|
data/limeta.gemspec
ADDED
@@ -0,0 +1,40 @@
|
|
1
|
+
lib = File.expand_path("../lib", __FILE__)
|
2
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
3
|
+
require "limeta/version"
|
4
|
+
|
5
|
+
Gem::Specification.new do |spec|
|
6
|
+
spec.name = "limeta"
|
7
|
+
spec.version = Limeta::VERSION
|
8
|
+
spec.authors = ["Edwin Onuonga"]
|
9
|
+
spec.email = ["ed@eonu.net"]
|
10
|
+
|
11
|
+
spec.summary = %q{Conversion tool for extracting chess game metadata from Lichess' public PGN files and exporting it to database tables.}
|
12
|
+
spec.homepage = "https://github.com/eonu/limeta"
|
13
|
+
|
14
|
+
spec.files = Dir.glob('lib/**/*', File::FNM_DOTMATCH) + %w[
|
15
|
+
Gemfile README.md Rakefile limeta.gemspec
|
16
|
+
]
|
17
|
+
spec.bindir = "bin"
|
18
|
+
spec.executables = "limeta"
|
19
|
+
spec.require_paths = ["lib"]
|
20
|
+
|
21
|
+
spec.add_runtime_dependency "thor", "~> 0.20"
|
22
|
+
spec.add_runtime_dependency "sequel", "~> 5.22"
|
23
|
+
spec.add_runtime_dependency "sqlite3", "~> 1.4"
|
24
|
+
spec.add_runtime_dependency "pg", ">= 1.1"
|
25
|
+
spec.add_runtime_dependency "pg-url", "~> 0.1", ">= 0.1.2"
|
26
|
+
spec.add_runtime_dependency "string-builder", "~> 2.3", ">= 2.3.1"
|
27
|
+
spec.add_runtime_dependency "tty-progressbar", "~> 0.17"
|
28
|
+
spec.add_runtime_dependency "tty-prompt", "~> 0.19"
|
29
|
+
|
30
|
+
spec.add_development_dependency "bundler", "~> 1.17"
|
31
|
+
spec.add_development_dependency "rake", "~> 12.3"
|
32
|
+
|
33
|
+
spec.metadata = {
|
34
|
+
'source_code_uri' => spec.homepage,
|
35
|
+
'homepage_uri' => spec.homepage,
|
36
|
+
'documentation_uri' => 'https://www.rubydoc.info/github/eonu/limeta/master/toplevel',
|
37
|
+
'bug_tracker_uri' => "#{spec.homepage}/issues",
|
38
|
+
'changelog_uri' => "#{spec.homepage}/blob/master/CHANGELOG.md"
|
39
|
+
}
|
40
|
+
end
|
metadata
ADDED
@@ -0,0 +1,211 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: limeta
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Edwin Onuonga
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2019-07-24 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: thor
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '0.20'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '0.20'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: sequel
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '5.22'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '5.22'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: sqlite3
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '1.4'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '1.4'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: pg
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '1.1'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '1.1'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: pg-url
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - "~>"
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0.1'
|
76
|
+
- - ">="
|
77
|
+
- !ruby/object:Gem::Version
|
78
|
+
version: 0.1.2
|
79
|
+
type: :runtime
|
80
|
+
prerelease: false
|
81
|
+
version_requirements: !ruby/object:Gem::Requirement
|
82
|
+
requirements:
|
83
|
+
- - "~>"
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: '0.1'
|
86
|
+
- - ">="
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: 0.1.2
|
89
|
+
- !ruby/object:Gem::Dependency
|
90
|
+
name: string-builder
|
91
|
+
requirement: !ruby/object:Gem::Requirement
|
92
|
+
requirements:
|
93
|
+
- - "~>"
|
94
|
+
- !ruby/object:Gem::Version
|
95
|
+
version: '2.3'
|
96
|
+
- - ">="
|
97
|
+
- !ruby/object:Gem::Version
|
98
|
+
version: 2.3.1
|
99
|
+
type: :runtime
|
100
|
+
prerelease: false
|
101
|
+
version_requirements: !ruby/object:Gem::Requirement
|
102
|
+
requirements:
|
103
|
+
- - "~>"
|
104
|
+
- !ruby/object:Gem::Version
|
105
|
+
version: '2.3'
|
106
|
+
- - ">="
|
107
|
+
- !ruby/object:Gem::Version
|
108
|
+
version: 2.3.1
|
109
|
+
- !ruby/object:Gem::Dependency
|
110
|
+
name: tty-progressbar
|
111
|
+
requirement: !ruby/object:Gem::Requirement
|
112
|
+
requirements:
|
113
|
+
- - "~>"
|
114
|
+
- !ruby/object:Gem::Version
|
115
|
+
version: '0.17'
|
116
|
+
type: :runtime
|
117
|
+
prerelease: false
|
118
|
+
version_requirements: !ruby/object:Gem::Requirement
|
119
|
+
requirements:
|
120
|
+
- - "~>"
|
121
|
+
- !ruby/object:Gem::Version
|
122
|
+
version: '0.17'
|
123
|
+
- !ruby/object:Gem::Dependency
|
124
|
+
name: tty-prompt
|
125
|
+
requirement: !ruby/object:Gem::Requirement
|
126
|
+
requirements:
|
127
|
+
- - "~>"
|
128
|
+
- !ruby/object:Gem::Version
|
129
|
+
version: '0.19'
|
130
|
+
type: :runtime
|
131
|
+
prerelease: false
|
132
|
+
version_requirements: !ruby/object:Gem::Requirement
|
133
|
+
requirements:
|
134
|
+
- - "~>"
|
135
|
+
- !ruby/object:Gem::Version
|
136
|
+
version: '0.19'
|
137
|
+
- !ruby/object:Gem::Dependency
|
138
|
+
name: bundler
|
139
|
+
requirement: !ruby/object:Gem::Requirement
|
140
|
+
requirements:
|
141
|
+
- - "~>"
|
142
|
+
- !ruby/object:Gem::Version
|
143
|
+
version: '1.17'
|
144
|
+
type: :development
|
145
|
+
prerelease: false
|
146
|
+
version_requirements: !ruby/object:Gem::Requirement
|
147
|
+
requirements:
|
148
|
+
- - "~>"
|
149
|
+
- !ruby/object:Gem::Version
|
150
|
+
version: '1.17'
|
151
|
+
- !ruby/object:Gem::Dependency
|
152
|
+
name: rake
|
153
|
+
requirement: !ruby/object:Gem::Requirement
|
154
|
+
requirements:
|
155
|
+
- - "~>"
|
156
|
+
- !ruby/object:Gem::Version
|
157
|
+
version: '12.3'
|
158
|
+
type: :development
|
159
|
+
prerelease: false
|
160
|
+
version_requirements: !ruby/object:Gem::Requirement
|
161
|
+
requirements:
|
162
|
+
- - "~>"
|
163
|
+
- !ruby/object:Gem::Version
|
164
|
+
version: '12.3'
|
165
|
+
description:
|
166
|
+
email:
|
167
|
+
- ed@eonu.net
|
168
|
+
executables:
|
169
|
+
- limeta
|
170
|
+
extensions: []
|
171
|
+
extra_rdoc_files: []
|
172
|
+
files:
|
173
|
+
- Gemfile
|
174
|
+
- README.md
|
175
|
+
- Rakefile
|
176
|
+
- bin/limeta
|
177
|
+
- lib/limeta.rb
|
178
|
+
- lib/limeta/cli.rb
|
179
|
+
- lib/limeta/connection.rb
|
180
|
+
- lib/limeta/table.rb
|
181
|
+
- lib/limeta/version.rb
|
182
|
+
- limeta.gemspec
|
183
|
+
homepage: https://github.com/eonu/limeta
|
184
|
+
licenses: []
|
185
|
+
metadata:
|
186
|
+
source_code_uri: https://github.com/eonu/limeta
|
187
|
+
homepage_uri: https://github.com/eonu/limeta
|
188
|
+
documentation_uri: https://www.rubydoc.info/github/eonu/limeta/master/toplevel
|
189
|
+
bug_tracker_uri: https://github.com/eonu/limeta/issues
|
190
|
+
changelog_uri: https://github.com/eonu/limeta/blob/master/CHANGELOG.md
|
191
|
+
post_install_message:
|
192
|
+
rdoc_options: []
|
193
|
+
require_paths:
|
194
|
+
- lib
|
195
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
196
|
+
requirements:
|
197
|
+
- - ">="
|
198
|
+
- !ruby/object:Gem::Version
|
199
|
+
version: '0'
|
200
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
201
|
+
requirements:
|
202
|
+
- - ">="
|
203
|
+
- !ruby/object:Gem::Version
|
204
|
+
version: '0'
|
205
|
+
requirements: []
|
206
|
+
rubygems_version: 3.0.3
|
207
|
+
signing_key:
|
208
|
+
specification_version: 4
|
209
|
+
summary: Conversion tool for extracting chess game metadata from Lichess' public PGN
|
210
|
+
files and exporting it to database tables.
|
211
|
+
test_files: []
|