mode 0.0.13 → 0.0.14
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +0 -1
- data/README.md +74 -129
- data/doc/install_ruby.md +65 -0
- data/doc/packages.md +94 -0
- data/lib/mode/config.rb +9 -1
- data/lib/mode/connector/data_source.rb +1 -1
- data/lib/mode/version.rb +1 -1
- data/spec/api/request_spec.rb +1 -1
- metadata +3 -1
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 306d6b69e36902cfcfbbb337bd5fcb2a9ab34885
|
4
|
+
data.tar.gz: 1e37f39443b33605cc75857b0cb29ea7a4d3f645
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: cc645a00b69a3fe58d1af485d1e538c6e19cfbf36f7ce4b4964fde02125ca0abbf72b64edeb7b5a32a1083c02d2b6152d39e5fde7aba53a7396b7eece18af562
|
7
|
+
data.tar.gz: 09acba6c1e90aad0627c1f71008e11da836a0fc83c02e847cef543c4c7f9bc6ef8310d76d8b544456eac0ec4051883cc17e453238bd1f90065581e16ba87263c
|
data/.gitignore
CHANGED
data/README.md
CHANGED
@@ -1,187 +1,132 @@
|
|
1
1
|
# Mode
|
2
2
|
|
3
|
-
This
|
4
|
-
|
5
|
-
* Cloud data warehouse management
|
6
|
-
* Personal data warehouse connectivity
|
7
|
-
* Dataset formatting and importing (CSV)
|
8
|
-
|
3
|
+
This gem provides command line tools for interacting with the Mode Analytics APIs.
|
9
4
|
|
10
5
|
## Install Mode
|
11
6
|
|
12
|
-
**Mode requires Ruby 1.9 or newer**
|
7
|
+
**Mode requires Ruby 1.9 or JRuby 1.7 or newer**
|
13
8
|
|
14
|
-
|
9
|
+
For the best performance and compatibility we recommend JRuby.
|
10
|
+
|
11
|
+
If you don't currently have a suitable ruby or aren't sure then follow [these directions](doc/install_ruby.md) to install it before continuing.
|
15
12
|
|
16
13
|
```
|
17
14
|
gem install mode
|
18
15
|
```
|
19
16
|
|
20
|
-
##
|
21
|
-
|
22
|
-
This package requires at least Ruby 1.9 and Ruby 2.0 is recommended.
|
17
|
+
## Logging In
|
23
18
|
|
24
|
-
|
19
|
+
Before you use the mode gem you'll need to login.
|
25
20
|
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
1\. Install Homebrew
|
21
|
+
The login process allows you to authorize the machine to connect via
|
22
|
+
the API with the use of an API access token. The first time you use this
|
23
|
+
command the it will look as follows:
|
31
24
|
|
32
25
|
```
|
33
|
-
|
34
|
-
|
26
|
+
$ mode login
|
27
|
+
Enter your Mode credentials:
|
28
|
+
Username: besquared
|
29
|
+
Password:
|
35
30
|
|
36
|
-
|
31
|
+
No API tokens found, would you like to create a new token? [y/n] y
|
37
32
|
|
38
|
-
|
39
|
-
|
40
|
-
brew install rbenv ruby-build rbenv-gem-rehash
|
41
|
-
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
|
42
|
-
source ~/.bash_profile
|
33
|
+
Choose a token name: Connector
|
34
|
+
Updated configuration at /Users/josh/.mode/config.yml
|
43
35
|
```
|
44
36
|
|
45
|
-
|
37
|
+
## Adding Data Sources
|
46
38
|
|
47
|
-
|
39
|
+
Open the Mode configuration file in your favorite text editor (I'm using sublime text)
|
48
40
|
|
49
41
|
```
|
50
|
-
|
51
|
-
rbenv global 2.0.0-p353
|
42
|
+
$ subl ~/.mode/config.yml
|
52
43
|
```
|
53
44
|
|
54
|
-
|
55
|
-
|
56
|
-
For convenience you can just copy and paste all the lines at once into your terminal
|
45
|
+
A sample data sources configuration is listed which you can copy to the bottom of your config file and modify:
|
57
46
|
|
58
47
|
```
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
```
|
67
|
-
|
68
|
-
### Windows
|
48
|
+
data_sources:
|
49
|
+
development:
|
50
|
+
adapter: jdbc:postgresql
|
51
|
+
host: localhost
|
52
|
+
database: development
|
53
|
+
username: postgres
|
54
|
+
password:
|
69
55
|
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
56
|
+
warehouse:
|
57
|
+
adapter: jdbc:redshift
|
58
|
+
host: hostname.redshift.amazonaws.com
|
59
|
+
port: 5439
|
60
|
+
database: dev
|
61
|
+
username:
|
62
|
+
password:
|
63
|
+
```
|
75
64
|
|
76
|
-
##
|
65
|
+
## Database Drivers
|
77
66
|
|
78
|
-
|
67
|
+
If you're using JRuby then you can load JDBC drivers by copying the driver's jar files into Mode's drivers directory
|
79
68
|
|
80
|
-
|
69
|
+
Example for teradata:
|
81
70
|
|
82
71
|
```
|
83
|
-
|
84
|
-
|
85
|
-
Initializing configuration at /Users/josh/.mode.yml
|
86
|
-
Mode username: besquared
|
87
|
-
Your can view your access tokens at http://www.modeanalytics.com/accounts/besquared/access_tokens
|
88
|
-
Access token for besquared: ...
|
89
|
-
Wrote configuration to /Users/josh/.mode.yml
|
72
|
+
cp terajdbc4.jar ~/mode/drivers
|
73
|
+
cp tdgssconfig.jar ~/mode/drivers
|
90
74
|
```
|
91
75
|
|
92
|
-
|
76
|
+
For ruby1.9+ you need to install the correct driver gem and libraries
|
93
77
|
|
94
|
-
|
78
|
+
```
|
79
|
+
# PostgreSQL
|
80
|
+
gem install pg
|
95
81
|
|
96
|
-
|
82
|
+
# Redshift
|
83
|
+
gem install sequel_redshift
|
97
84
|
|
98
|
-
|
99
|
-
|
85
|
+
# MySQL
|
86
|
+
gem install mysql2
|
100
87
|
|
101
|
-
|
88
|
+
# Oracle 9i+
|
89
|
+
gem install oci8
|
102
90
|
|
103
|
-
|
91
|
+
# SQL Server & Azure
|
92
|
+
gem install tiny_tds
|
104
93
|
|
94
|
+
# DB2
|
95
|
+
gem install ibm_db
|
105
96
|
```
|
106
|
-
$ mode analyze bikeshare_small.csv
|
107
|
-
|
108
|
-
# Analyzing bikeshare_small.csv (Sampling 12.70%)...
|
109
|
-
# Analyzed 12040 of 99999 rows
|
110
|
-
+------------+-----+----------+------------+-------------+------------+---------------+-------------+-----------+
|
111
|
-
| Field | Key | Type | String (%) | Integer (%) | Number (%) | Date/Time (%) | Boolean (%) | Empty (%) |
|
112
|
-
+------------+-----+----------+------------+-------------+------------+---------------+-------------+-----------+
|
113
|
-
| start_time | No | datetime | | | | 100.00% | | |
|
114
|
-
| type | No | string | 100.00% | | | | | |
|
115
|
-
+------------+-----+----------+------------+-------------+------------+---------------+-------------+-----------+
|
116
|
-
```
|
117
|
-
|
118
|
-
### Import
|
119
97
|
|
120
|
-
|
98
|
+
## Verifying your API and driver configuration
|
121
99
|
|
122
|
-
|
123
|
-
|
124
|
-
The package command will analyze your dataset and create a data package based on the field types that are recognized.
|
125
|
-
|
126
|
-
The data package can then be imported into the data warehouse.
|
100
|
+
You can verify that your data sources are correct by using the verify command:
|
127
101
|
|
128
102
|
```
|
103
|
+
$ mode connect verify
|
129
104
|
```
|
130
105
|
|
131
|
-
|
132
|
-
|
133
|
-
Mode packages are inspired by and compatible with [Open Knowledge Foundation Standards](http://data.okfn.org/standards), and specifically:
|
134
|
-
|
135
|
-
* [Data Packages](http://data.okfn.org/standards/data-package)
|
136
|
-
* [Simple Data Format](http://data.okfn.org/standards/simple-data-format)
|
106
|
+
## Controlling the connector
|
137
107
|
|
138
|
-
|
108
|
+
Starting
|
139
109
|
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
Additionally they may contain a README.md, scripts and other resources.
|
144
|
-
|
145
|
-
#### datapackage.json
|
110
|
+
```
|
111
|
+
$ mode connect start
|
112
|
+
```
|
146
113
|
|
147
|
-
|
114
|
+
Restarting
|
148
115
|
|
149
116
|
```
|
150
|
-
|
151
|
-
|
152
|
-
"version": 0.0.1,
|
153
|
-
"title": "a human friendly title"
|
117
|
+
$ mode connect restart
|
118
|
+
```
|
154
119
|
|
155
|
-
|
156
|
-
{
|
157
|
-
"name": "data",
|
158
|
-
"format": "csv",
|
159
|
-
"path": "data.csv",
|
120
|
+
Stopping
|
160
121
|
|
161
|
-
|
162
|
-
|
163
|
-
|
164
|
-
"doubleQuote": false,
|
165
|
-
"lineTerminator": "\r\n",
|
166
|
-
"skipInitialSpace": false
|
167
|
-
},
|
122
|
+
```
|
123
|
+
$ mode connect stop
|
124
|
+
```
|
168
125
|
|
169
|
-
|
170
|
-
"fields": [
|
171
|
-
{
|
172
|
-
"name": "name of field (e.g. column name)",
|
173
|
-
"title": "A nicer human readable label or title for the field",
|
174
|
-
"type": "A string specifying the type",
|
175
|
-
"format": "A string specifying a format",
|
176
|
-
"description": "A description for the field"
|
177
|
-
...
|
178
|
-
},
|
179
|
-
],
|
126
|
+
## Connector logging
|
180
127
|
|
181
|
-
|
182
|
-
}
|
183
|
-
}
|
184
|
-
]
|
185
|
-
}
|
128
|
+
The connect logs all its activity and errors to a log file that rotates weekly by default.
|
186
129
|
|
130
|
+
```
|
131
|
+
tail -f ~/.mode/log/mode.log
|
187
132
|
```
|
data/doc/install_ruby.md
ADDED
@@ -0,0 +1,65 @@
|
|
1
|
+
|
2
|
+
## Install Ruby 1.9+ if it's not currently installed.
|
3
|
+
|
4
|
+
This package requires at least Ruby 1.9 and Ruby 2.0 is recommended.
|
5
|
+
|
6
|
+
If you don't have an up to date version of Ruby or you're not sure then use the instructions below to get going.
|
7
|
+
|
8
|
+
### Mac OSX
|
9
|
+
|
10
|
+
To install Ruby on OSX complete the 4 steps below.
|
11
|
+
|
12
|
+
1\. Install Homebrew
|
13
|
+
|
14
|
+
```
|
15
|
+
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
|
16
|
+
```
|
17
|
+
|
18
|
+
2\. Install rbenv
|
19
|
+
|
20
|
+
```
|
21
|
+
brew update
|
22
|
+
brew install rbenv ruby-build rbenv-gem-rehash
|
23
|
+
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
|
24
|
+
source ~/.bash_profile
|
25
|
+
```
|
26
|
+
|
27
|
+
3\. Install Ruby or JRuby
|
28
|
+
|
29
|
+
Note: This usually takes several minutes
|
30
|
+
|
31
|
+
Ruby
|
32
|
+
|
33
|
+
```
|
34
|
+
rbenv install 2.0.0-p353
|
35
|
+
rbenv global 2.0.0-p353
|
36
|
+
```
|
37
|
+
|
38
|
+
JRuby
|
39
|
+
|
40
|
+
```
|
41
|
+
rbenv install jruby-1.7.9
|
42
|
+
rbenv global jruby-1.7.9
|
43
|
+
```
|
44
|
+
|
45
|
+
### OSX Combined
|
46
|
+
|
47
|
+
For convenience you can just copy and paste all the lines at once into your terminal
|
48
|
+
|
49
|
+
```
|
50
|
+
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
|
51
|
+
brew update
|
52
|
+
brew install rbenv ruby-build rbenv-gem-rehash
|
53
|
+
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
|
54
|
+
source ~/.bash_profile
|
55
|
+
rbenv install 2.0.0-p353
|
56
|
+
rbenv global 2.0.0-p353
|
57
|
+
```
|
58
|
+
|
59
|
+
### Windows
|
60
|
+
|
61
|
+
To install a current version of Ruby on windows complete the 2 steps below.
|
62
|
+
|
63
|
+
1\. Install Ruby
|
64
|
+
|
65
|
+
You can download the latest ruby version from [RubyInstaller](http://rubyinstaller.org/downloads/).
|
data/doc/packages.md
ADDED
@@ -0,0 +1,94 @@
|
|
1
|
+
## Working with CSV Data
|
2
|
+
|
3
|
+
### Analyze
|
4
|
+
|
5
|
+
The analyze command performs two useful functions:
|
6
|
+
|
7
|
+
* Ensures that the csv is well formed with no syntax errors
|
8
|
+
* Tells you how Mode will recognize data types and format data
|
9
|
+
|
10
|
+
For better performance analyze only inspects a sampled subset of file.
|
11
|
+
|
12
|
+
You can optionally set the sampling rate by passing the --sample option with a number between 0 and 1.
|
13
|
+
|
14
|
+
```
|
15
|
+
$ mode analyze bikeshare_small.csv
|
16
|
+
|
17
|
+
# Analyzing bikeshare_small.csv (Sampling 12.70%)...
|
18
|
+
# Analyzed 12040 of 99999 rows
|
19
|
+
+------------+-----+----------+------------+-------------+------------+---------------+-------------+-----------+
|
20
|
+
| Field | Key | Type | String (%) | Integer (%) | Number (%) | Date/Time (%) | Boolean (%) | Empty (%) |
|
21
|
+
+------------+-----+----------+------------+-------------+------------+---------------+-------------+-----------+
|
22
|
+
| start_time | No | datetime | | | | 100.00% | | |
|
23
|
+
| type | No | string | 100.00% | | | | | |
|
24
|
+
+------------+-----+----------+------------+-------------+------------+---------------+-------------+-----------+
|
25
|
+
```
|
26
|
+
|
27
|
+
### Import
|
28
|
+
|
29
|
+
More to come!
|
30
|
+
|
31
|
+
### Package
|
32
|
+
|
33
|
+
The package command will analyze your dataset and create a data package based on the field types that are recognized.
|
34
|
+
|
35
|
+
The data package can then be imported into the data warehouse.
|
36
|
+
|
37
|
+
```
|
38
|
+
```
|
39
|
+
|
40
|
+
### Package Format
|
41
|
+
|
42
|
+
Mode packages are inspired by and compatible with [Open Knowledge Foundation Standards](http://data.okfn.org/standards), and specifically:
|
43
|
+
|
44
|
+
* [Data Packages](http://data.okfn.org/standards/data-package)
|
45
|
+
* [Simple Data Format](http://data.okfn.org/standards/simple-data-format)
|
46
|
+
|
47
|
+
The most basic Mode data packages contains a normalized csv data file as well as a file describing the data.
|
48
|
+
|
49
|
+
* data.csv
|
50
|
+
* datapackage.json
|
51
|
+
|
52
|
+
Additionally they may contain a README.md, scripts and other resources.
|
53
|
+
|
54
|
+
#### datapackage.json
|
55
|
+
|
56
|
+
The datapackage.json file contains descriptive information of the data available in the package including naming, formatting and schema information.
|
57
|
+
|
58
|
+
```
|
59
|
+
{
|
60
|
+
"name" : "my-dataset",
|
61
|
+
"version": 0.0.1,
|
62
|
+
"title": "a human friendly title"
|
63
|
+
|
64
|
+
"resources": [
|
65
|
+
{
|
66
|
+
"name": "data",
|
67
|
+
"format": "csv",
|
68
|
+
"path": "data.csv",
|
69
|
+
|
70
|
+
"dialect": {
|
71
|
+
"delimiter": ",",
|
72
|
+
"quoteChar": "\"",
|
73
|
+
"doubleQuote": false,
|
74
|
+
"lineTerminator": "\r\n",
|
75
|
+
"skipInitialSpace": false
|
76
|
+
},
|
77
|
+
|
78
|
+
"schema": {
|
79
|
+
"fields": [
|
80
|
+
{
|
81
|
+
"name": "name of field (e.g. column name)",
|
82
|
+
"title": "A nicer human readable label or title for the field",
|
83
|
+
"type": "A string specifying the type",
|
84
|
+
"format": "A string specifying a format",
|
85
|
+
"description": "A description for the field"
|
86
|
+
...
|
87
|
+
},
|
88
|
+
],
|
89
|
+
|
90
|
+
"primaryKey": ['field1', 'field2', ...]
|
91
|
+
}
|
92
|
+
}
|
93
|
+
]
|
94
|
+
}
|
data/lib/mode/config.rb
CHANGED
@@ -86,12 +86,20 @@ module Mode
|
|
86
86
|
end
|
87
87
|
end
|
88
88
|
|
89
|
+
def data_source_hash
|
90
|
+
hsh = {}
|
91
|
+
data_sources.each do |data_source|
|
92
|
+
hsh[data_source.name] = data_source.props
|
93
|
+
end
|
94
|
+
hsh
|
95
|
+
end
|
96
|
+
|
89
97
|
def to_yaml
|
90
98
|
{
|
91
99
|
'username' => username,
|
92
100
|
'access_token' => access_token,
|
93
101
|
'environment' => environment,
|
94
|
-
'data_sources' =>
|
102
|
+
'data_sources' => data_source_hash
|
95
103
|
}.to_yaml
|
96
104
|
end
|
97
105
|
end
|
data/lib/mode/version.rb
CHANGED
data/spec/api/request_spec.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: mode
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.14
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Mode Analytics
|
@@ -236,6 +236,8 @@ files:
|
|
236
236
|
- README.md
|
237
237
|
- Rakefile
|
238
238
|
- bin/mode
|
239
|
+
- doc/install_ruby.md
|
240
|
+
- doc/packages.md
|
239
241
|
- lib/connect.rb
|
240
242
|
- lib/mode.rb
|
241
243
|
- lib/mode/api/form.rb
|