blazer 2.2.6
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of blazer might be problematic. Click here for more details.
- checksums.yaml +7 -0
- data/CHANGELOG.md +310 -0
- data/CONTRIBUTING.md +42 -0
- data/LICENSE.txt +22 -0
- data/README.md +1041 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.eot +0 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.svg +288 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.ttf +0 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.woff +0 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.woff2 +0 -0
- data/app/assets/images/blazer/favicon.png +0 -0
- data/app/assets/javascripts/blazer/Chart.js +14456 -0
- data/app/assets/javascripts/blazer/Sortable.js +1540 -0
- data/app/assets/javascripts/blazer/ace.js +6 -0
- data/app/assets/javascripts/blazer/ace/ace.js +21301 -0
- data/app/assets/javascripts/blazer/ace/ext-language_tools.js +1993 -0
- data/app/assets/javascripts/blazer/ace/mode-sql.js +110 -0
- data/app/assets/javascripts/blazer/ace/snippets/sql.js +40 -0
- data/app/assets/javascripts/blazer/ace/snippets/text.js +14 -0
- data/app/assets/javascripts/blazer/ace/theme-twilight.js +116 -0
- data/app/assets/javascripts/blazer/application.js +81 -0
- data/app/assets/javascripts/blazer/bootstrap.js +2377 -0
- data/app/assets/javascripts/blazer/chartkick.js +2214 -0
- data/app/assets/javascripts/blazer/daterangepicker.js +1653 -0
- data/app/assets/javascripts/blazer/fuzzysearch.js +24 -0
- data/app/assets/javascripts/blazer/highlight.min.js +3 -0
- data/app/assets/javascripts/blazer/jquery-ujs.js +555 -0
- data/app/assets/javascripts/blazer/jquery.js +10364 -0
- data/app/assets/javascripts/blazer/jquery.stickytableheaders.js +325 -0
- data/app/assets/javascripts/blazer/moment-timezone-with-data.js +1212 -0
- data/app/assets/javascripts/blazer/moment.js +3043 -0
- data/app/assets/javascripts/blazer/queries.js +110 -0
- data/app/assets/javascripts/blazer/routes.js +26 -0
- data/app/assets/javascripts/blazer/selectize.js +3891 -0
- data/app/assets/javascripts/blazer/stupidtable-custom-settings.js +13 -0
- data/app/assets/javascripts/blazer/stupidtable.js +281 -0
- data/app/assets/javascripts/blazer/vue.js +10947 -0
- data/app/assets/stylesheets/blazer/application.css +234 -0
- data/app/assets/stylesheets/blazer/bootstrap.css.erb +6756 -0
- data/app/assets/stylesheets/blazer/daterangepicker.css +269 -0
- data/app/assets/stylesheets/blazer/github.css +125 -0
- data/app/assets/stylesheets/blazer/selectize.css +403 -0
- data/app/controllers/blazer/base_controller.rb +124 -0
- data/app/controllers/blazer/checks_controller.rb +56 -0
- data/app/controllers/blazer/dashboards_controller.rb +101 -0
- data/app/controllers/blazer/queries_controller.rb +347 -0
- data/app/helpers/blazer/base_helper.rb +43 -0
- data/app/mailers/blazer/check_mailer.rb +27 -0
- data/app/mailers/blazer/slack_notifier.rb +79 -0
- data/app/models/blazer/audit.rb +6 -0
- data/app/models/blazer/check.rb +104 -0
- data/app/models/blazer/connection.rb +5 -0
- data/app/models/blazer/dashboard.rb +17 -0
- data/app/models/blazer/dashboard_query.rb +9 -0
- data/app/models/blazer/query.rb +40 -0
- data/app/models/blazer/record.rb +5 -0
- data/app/views/blazer/_nav.html.erb +15 -0
- data/app/views/blazer/_variables.html.erb +124 -0
- data/app/views/blazer/check_mailer/failing_checks.html.erb +7 -0
- data/app/views/blazer/check_mailer/state_change.html.erb +48 -0
- data/app/views/blazer/checks/_form.html.erb +79 -0
- data/app/views/blazer/checks/edit.html.erb +3 -0
- data/app/views/blazer/checks/index.html.erb +69 -0
- data/app/views/blazer/checks/new.html.erb +3 -0
- data/app/views/blazer/dashboards/_form.html.erb +76 -0
- data/app/views/blazer/dashboards/edit.html.erb +3 -0
- data/app/views/blazer/dashboards/new.html.erb +3 -0
- data/app/views/blazer/dashboards/show.html.erb +51 -0
- data/app/views/blazer/queries/_form.html.erb +250 -0
- data/app/views/blazer/queries/docs.html.erb +131 -0
- data/app/views/blazer/queries/edit.html.erb +2 -0
- data/app/views/blazer/queries/home.html.erb +163 -0
- data/app/views/blazer/queries/new.html.erb +2 -0
- data/app/views/blazer/queries/run.html.erb +198 -0
- data/app/views/blazer/queries/schema.html.erb +55 -0
- data/app/views/blazer/queries/show.html.erb +75 -0
- data/app/views/layouts/blazer/application.html.erb +24 -0
- data/config/routes.rb +20 -0
- data/lib/blazer.rb +231 -0
- data/lib/blazer/adapters/athena_adapter.rb +129 -0
- data/lib/blazer/adapters/base_adapter.rb +53 -0
- data/lib/blazer/adapters/bigquery_adapter.rb +68 -0
- data/lib/blazer/adapters/cassandra_adapter.rb +59 -0
- data/lib/blazer/adapters/drill_adapter.rb +28 -0
- data/lib/blazer/adapters/druid_adapter.rb +67 -0
- data/lib/blazer/adapters/elasticsearch_adapter.rb +46 -0
- data/lib/blazer/adapters/influxdb_adapter.rb +45 -0
- data/lib/blazer/adapters/mongodb_adapter.rb +39 -0
- data/lib/blazer/adapters/neo4j_adapter.rb +47 -0
- data/lib/blazer/adapters/presto_adapter.rb +45 -0
- data/lib/blazer/adapters/salesforce_adapter.rb +45 -0
- data/lib/blazer/adapters/snowflake_adapter.rb +73 -0
- data/lib/blazer/adapters/soda_adapter.rb +96 -0
- data/lib/blazer/adapters/sql_adapter.rb +221 -0
- data/lib/blazer/data_source.rb +195 -0
- data/lib/blazer/detect_anomalies.R +19 -0
- data/lib/blazer/engine.rb +43 -0
- data/lib/blazer/result.rb +218 -0
- data/lib/blazer/run_statement.rb +40 -0
- data/lib/blazer/run_statement_job.rb +18 -0
- data/lib/blazer/version.rb +3 -0
- data/lib/generators/blazer/install_generator.rb +22 -0
- data/lib/generators/blazer/templates/config.yml.tt +73 -0
- data/lib/generators/blazer/templates/install.rb.tt +46 -0
- data/lib/tasks/blazer.rake +11 -0
- metadata +231 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: cfb774d21d8d5756c2fc273c2148542e26550452ea886adf92023c411b45200a
|
4
|
+
data.tar.gz: 8200979166341f38e0647fbb566f1fd8ead6895ea254a12ead4901e344d80f38
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: d6705d3dd2c27db9aeb36b9afd2e9a335d876f36865c5bbaade10f13cca80436ec3c31787231c16b591fd78481cdbb7f11df82de00417fd71d5005b1db038b11
|
7
|
+
data.tar.gz: 50c2230e80ad2a0e487197c8593a4c774a0f58304ed88b704860ce4550029202f677042a685b7457a57c67b6e660a82f4692ba529a5eabb41c17d2d1182c3fce
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,310 @@
|
|
1
|
+
## 2.2.6 (2020-07-21)
|
2
|
+
|
3
|
+
- Added experimental support for InfluxDB
|
4
|
+
- Added support for forecasting week, month, quarter, and year with Prophet
|
5
|
+
- Fixed forecasting link not showing up
|
6
|
+
|
7
|
+
## 2.2.5 (2020-06-03)
|
8
|
+
|
9
|
+
- Updated maps to fix deprecation error
|
10
|
+
|
11
|
+
## 2.2.4 (2020-05-30)
|
12
|
+
|
13
|
+
- Fixed error with new queries
|
14
|
+
|
15
|
+
## 2.2.3 (2020-05-30)
|
16
|
+
|
17
|
+
- Improved query parameter handling
|
18
|
+
|
19
|
+
## 2.2.2 (2020-04-13)
|
20
|
+
|
21
|
+
- Added experimental support for the Socrata Open Data API (SODA)
|
22
|
+
- Added experimental Prophet forecasting
|
23
|
+
- Fixed query search for non-ASCII characters
|
24
|
+
|
25
|
+
## 2.2.1 (2019-10-08)
|
26
|
+
|
27
|
+
- Added support for Sprockets 4
|
28
|
+
- Improved Snowflake table preview
|
29
|
+
- Fixed bug with refresh link not showing
|
30
|
+
|
31
|
+
## 2.2.0 (2019-07-12)
|
32
|
+
|
33
|
+
- Added schema to table preview for Postgres and Redshift
|
34
|
+
- Fixed bug with Slack notifications not sending
|
35
|
+
- Dropped support for Rails 4.2
|
36
|
+
|
37
|
+
## 2.1.0 (2019-06-04)
|
38
|
+
|
39
|
+
- Require latest Chartkick to prevent possible XSS - see [#245](https://github.com/ankane/blazer/issues/245)
|
40
|
+
|
41
|
+
## 2.0.2 (2019-05-26)
|
42
|
+
|
43
|
+
- Added support for variable transformation for blind indexing
|
44
|
+
- Added experimental support for Neo4j
|
45
|
+
- Added experimental support for Salesforce
|
46
|
+
- Fixed JavaScript sorting for numbers with commas
|
47
|
+
|
48
|
+
## 2.0.1 (2019-01-07)
|
49
|
+
|
50
|
+
- Added favicon
|
51
|
+
- Added search for checks and schema
|
52
|
+
- Added pie charts
|
53
|
+
- Added Trend anomaly detection
|
54
|
+
- Added forecasting
|
55
|
+
- Improved tooltips
|
56
|
+
- Improved docs for new installs
|
57
|
+
- Fixed error with canceling queries
|
58
|
+
|
59
|
+
## 2.0.0 (2019-01-03)
|
60
|
+
|
61
|
+
- Added support for Slack
|
62
|
+
- Added `async` option
|
63
|
+
- Added `override_csp` option
|
64
|
+
- Added smart variables, linked columns smart columns, and charts to inline docs
|
65
|
+
- Use SQL for Elasticsearch
|
66
|
+
- Fixed error with latest `google-cloud-bigquery`
|
67
|
+
|
68
|
+
Breaking changes
|
69
|
+
|
70
|
+
- Dropped support for Rails < 4.2
|
71
|
+
|
72
|
+
## 1.9.0 (2018-06-17)
|
73
|
+
|
74
|
+
- Prompt developers to check custom `before_action`
|
75
|
+
- Better ordering on home page
|
76
|
+
- Added support for Snowflake
|
77
|
+
|
78
|
+
## 1.8.2 (2018-02-22)
|
79
|
+
|
80
|
+
- Added support for Cassandra
|
81
|
+
- Fixes for Druid
|
82
|
+
- Added support for Amazon Athena
|
83
|
+
- Added support for Druid
|
84
|
+
- Fixed query cancellation
|
85
|
+
|
86
|
+
There was no 1.8.1 release.
|
87
|
+
|
88
|
+
## 1.8.0 (2017-05-01)
|
89
|
+
|
90
|
+
- Added support for Rails 5.1
|
91
|
+
|
92
|
+
## 1.7.10 (2017-04-03)
|
93
|
+
|
94
|
+
- Added support for Google BigQuery
|
95
|
+
- Require `drill-sergeant` gem for Apache Drill
|
96
|
+
- Better handling of checks with variables
|
97
|
+
|
98
|
+
## 1.7.9 (2017-03-20)
|
99
|
+
|
100
|
+
- Added beta support for Apache Drill
|
101
|
+
- Added email validation for checks
|
102
|
+
- Updated Chart.js to 2.5.0
|
103
|
+
|
104
|
+
## 1.7.8 (2017-02-06)
|
105
|
+
|
106
|
+
- Added support for custom adapters
|
107
|
+
- Fixed bug with scatter charts on dashboards
|
108
|
+
- Fixed table preview for SQL Server
|
109
|
+
- Fixed issue when `default_url_options` set
|
110
|
+
|
111
|
+
## 1.7.7 (2016-12-17)
|
112
|
+
|
113
|
+
- Fixed preview error for MySQL
|
114
|
+
- Fixed error with timeouts for MySQL
|
115
|
+
|
116
|
+
## 1.7.6 (2016-12-13)
|
117
|
+
|
118
|
+
- Added scatter chart
|
119
|
+
- Fixed issue with false values showing up blank
|
120
|
+
- Fixed preview for table names with certain characters
|
121
|
+
|
122
|
+
## 1.7.5 (2016-11-22)
|
123
|
+
|
124
|
+
- Fixed issue with check emails sometimes failing for default Rails 5 ActiveJob adapter
|
125
|
+
- Fixed sorting for new dashboards
|
126
|
+
|
127
|
+
## 1.7.4 (2016-11-06)
|
128
|
+
|
129
|
+
- Removed extra dependencies added in 1.7.1
|
130
|
+
- Fixed `send_failing_checks` for default Rails 5 ActiveJob adapter
|
131
|
+
|
132
|
+
## 1.7.3 (2016-11-01)
|
133
|
+
|
134
|
+
- Fixed JavaScript errors
|
135
|
+
- Fixed query cancel error
|
136
|
+
- Return search results for "me" or "mine"
|
137
|
+
- Include sample data in email when bad data checks fail
|
138
|
+
- Fixed deprecation warnings
|
139
|
+
|
140
|
+
## 1.7.2 (2016-10-30)
|
141
|
+
|
142
|
+
- Cancel all queries on page nav
|
143
|
+
- Prevent Ace from taking over find command
|
144
|
+
- Added ability to use hashes for smart columns
|
145
|
+
- Added ability to inherit smart variables and columns from other data sources
|
146
|
+
|
147
|
+
## 1.7.1 (2016-10-29)
|
148
|
+
|
149
|
+
- Do not fork when enter key pressed
|
150
|
+
- Use custom version of Chart.js to fix label overlap
|
151
|
+
- Improved performance of home page
|
152
|
+
|
153
|
+
## 1.7.0 (2016-09-07)
|
154
|
+
|
155
|
+
- Added ability to cancel queries on backend for Postgres and Redshift
|
156
|
+
- Only run 3 queries at a time on dashboards
|
157
|
+
- Better anomaly detection
|
158
|
+
- Attempt to reconnect when connection issues
|
159
|
+
- Fixed issues with caching
|
160
|
+
|
161
|
+
## 1.6.2 (2016-08-11)
|
162
|
+
|
163
|
+
- Added basic query permissions
|
164
|
+
- Added ability to use arrays and hashes for smart variables
|
165
|
+
- Added cancel button for queries
|
166
|
+
- Added `lat` and `lng` as map keys
|
167
|
+
|
168
|
+
## 1.6.1 (2016-07-30)
|
169
|
+
|
170
|
+
- Added support for Presto [beta]
|
171
|
+
- Added support for Elasticsearch timeouts
|
172
|
+
- Fixed error in Rails 5
|
173
|
+
|
174
|
+
## 1.6.0 (2016-07-28)
|
175
|
+
|
176
|
+
- Added support for MongoDB [beta]
|
177
|
+
- Added support for Elasticsearch [beta]
|
178
|
+
- Fixed deprecation warning in Rails 5
|
179
|
+
|
180
|
+
## 1.5.1 (2016-07-24)
|
181
|
+
|
182
|
+
- Added anomaly detection for data less than 2 weeks
|
183
|
+
- Added autolinking urls
|
184
|
+
- Added support for images
|
185
|
+
|
186
|
+
## 1.5.0 (2016-06-29)
|
187
|
+
|
188
|
+
- Added new bar chart format
|
189
|
+
- Added anomaly detection checks
|
190
|
+
- Added `async` option for polling
|
191
|
+
|
192
|
+
## 1.4.0 (2016-06-09)
|
193
|
+
|
194
|
+
- Added `slow` cache mode
|
195
|
+
- Fixed `BLAZER_DATABASE_URL required` error
|
196
|
+
- Fixed issue with duplicate column names
|
197
|
+
|
198
|
+
## 1.3.5 (2016-05-11)
|
199
|
+
|
200
|
+
- Fixed error with checks
|
201
|
+
|
202
|
+
## 1.3.4 (2016-05-11)
|
203
|
+
|
204
|
+
- Fixed issue with missing queries
|
205
|
+
|
206
|
+
## 1.3.3 (2016-05-08)
|
207
|
+
|
208
|
+
- Fixed error with Rails 4.1 and below
|
209
|
+
|
210
|
+
## 1.3.2 (2016-05-07)
|
211
|
+
|
212
|
+
- Added support for Rails 5
|
213
|
+
- Attempt to reconnect for checks
|
214
|
+
|
215
|
+
## 1.3.1 (2016-05-06)
|
216
|
+
|
217
|
+
- Fixed migration error
|
218
|
+
|
219
|
+
## 1.3.0 (2016-05-06)
|
220
|
+
|
221
|
+
- Added schedule for checks
|
222
|
+
- Switched to Chart.js for charts
|
223
|
+
- Better output for explain
|
224
|
+
- Support for MySQL timeouts
|
225
|
+
- Raise error when timeout not supported
|
226
|
+
- Added creator to dashboards and checks
|
227
|
+
|
228
|
+
## 1.2.1 (2016-04-26)
|
229
|
+
|
230
|
+
- Fixed checks
|
231
|
+
|
232
|
+
## 1.2.0 (2016-03-22)
|
233
|
+
|
234
|
+
- Added non-editable queries
|
235
|
+
- Added variable defaults
|
236
|
+
- Added `local_time_suffix` setting
|
237
|
+
- Better timeout message
|
238
|
+
- Hide variables from commented out lines
|
239
|
+
- Fixed regex as variable names
|
240
|
+
|
241
|
+
## 1.1.1 (2016-03-06)
|
242
|
+
|
243
|
+
- Added `before_action` option
|
244
|
+
- Added invert option for checks
|
245
|
+
- Added targets
|
246
|
+
- Friendlier error message for timeouts
|
247
|
+
- Fixed request URI too large
|
248
|
+
- Prevent accidental backspace nav on query page
|
249
|
+
|
250
|
+
## 1.1.0 (2015-12-27)
|
251
|
+
|
252
|
+
- Replaced pie charts with column charts
|
253
|
+
- Fixed error with datepicker
|
254
|
+
- Added fork button to edit query page
|
255
|
+
- Added a notice when editing a query that is part of a dashboard
|
256
|
+
- Added refresh for dashboards
|
257
|
+
|
258
|
+
## 1.0.4 (2015-11-04)
|
259
|
+
|
260
|
+
- Added recently viewed queries and dashboards to home page
|
261
|
+
- Fixed refresh when transform statement is used
|
262
|
+
- Fixed error when no user model
|
263
|
+
|
264
|
+
## 1.0.3 (2015-10-18)
|
265
|
+
|
266
|
+
- Added maps
|
267
|
+
- Added support for Rails 4.0
|
268
|
+
|
269
|
+
## 1.0.2 (2015-10-11)
|
270
|
+
|
271
|
+
- Fixed error when installing
|
272
|
+
- Added `schemas` option
|
273
|
+
|
274
|
+
## 1.0.1 (2015-10-08)
|
275
|
+
|
276
|
+
- Added comments to queries
|
277
|
+
- Added `cache` option
|
278
|
+
- Added `user_method` option
|
279
|
+
- Added `use_transaction` option
|
280
|
+
|
281
|
+
## 1.0.0 (2015-10-04)
|
282
|
+
|
283
|
+
- Added support for multiple data sources
|
284
|
+
- Added dashboards
|
285
|
+
- Added checks
|
286
|
+
- Added support for Redshift
|
287
|
+
|
288
|
+
## 0.0.8 (2015-09-05)
|
289
|
+
|
290
|
+
- Easier to edit queries with variables
|
291
|
+
- Dynamically expand editor height as needed
|
292
|
+
- No need for spaces in search
|
293
|
+
|
294
|
+
## 0.0.7 (2015-07-23)
|
295
|
+
|
296
|
+
- Fixed error when no `User` class
|
297
|
+
- Fixed forking a query with variables
|
298
|
+
- Set time zone after Rails initializes
|
299
|
+
|
300
|
+
## 0.0.6 (2015-06-18)
|
301
|
+
|
302
|
+
- Added fork button
|
303
|
+
- Fixed trending
|
304
|
+
- Fixed time zones for date select
|
305
|
+
|
306
|
+
## 0.0.5 (2015-01-31)
|
307
|
+
|
308
|
+
- Added support for Rails 4.2
|
309
|
+
- Fixed error with `mysql2` adapter
|
310
|
+
- Added `user_class` option
|
data/CONTRIBUTING.md
ADDED
@@ -0,0 +1,42 @@
|
|
1
|
+
# Contributing
|
2
|
+
|
3
|
+
First, thanks for wanting to contribute. You’re awesome! :heart:
|
4
|
+
|
5
|
+
## Help
|
6
|
+
|
7
|
+
We’re not able to provide support through GitHub Issues. If you’re looking for help with your code, try posting on [Stack Overflow](https://stackoverflow.com/).
|
8
|
+
|
9
|
+
All features should be documented. If you don’t see a feature in the docs, assume it doesn’t exist.
|
10
|
+
|
11
|
+
## Bugs
|
12
|
+
|
13
|
+
Think you’ve discovered a bug?
|
14
|
+
|
15
|
+
1. Search existing issues to see if it’s been reported.
|
16
|
+
2. Try the `master` branch to make sure it hasn’t been fixed.
|
17
|
+
|
18
|
+
```rb
|
19
|
+
gem "blazer", github: "ankane/blazer"
|
20
|
+
```
|
21
|
+
|
22
|
+
If the above steps don’t help, create an issue. Include:
|
23
|
+
|
24
|
+
- Detailed steps to reproduce
|
25
|
+
- Complete backtraces for exceptions
|
26
|
+
|
27
|
+
## New Features
|
28
|
+
|
29
|
+
If you’d like to discuss a new feature, create an issue and start the title with `[Idea]`.
|
30
|
+
|
31
|
+
## Pull Requests
|
32
|
+
|
33
|
+
Fork the project and create a pull request. A few tips:
|
34
|
+
|
35
|
+
- Keep changes to a minimum. If you have multiple features or fixes, submit multiple pull requests.
|
36
|
+
- Follow the existing style. The code should read like it’s written by a single person.
|
37
|
+
|
38
|
+
Feel free to open an issue to get feedback on your idea before spending too much time on it.
|
39
|
+
|
40
|
+
---
|
41
|
+
|
42
|
+
This contributing guide is released under [CCO](https://creativecommons.org/publicdomain/zero/1.0/) (public domain). Use it for your own project without attribution.
|
data/LICENSE.txt
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2014-2019 Andrew Kane
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,1041 @@
|
|
1
|
+
# Blazer
|
2
|
+
|
3
|
+
Explore your data with SQL. Easily create charts and dashboards, and share them with your team.
|
4
|
+
|
5
|
+
[Try it out](https://blazer.dokkuapp.com)
|
6
|
+
|
7
|
+
[![Screenshot](https://blazer.dokkuapp.com/assets/blazer-90fb6ce8ad25614c6f9c3b4619cbfbd66fa3d6567dac34d83df540f6688665f1.png)](https://blazer.dokkuapp.com)
|
8
|
+
|
9
|
+
Blazer is also available as a [Docker image](https://github.com/ankane/blazer-docker).
|
10
|
+
|
11
|
+
:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
|
12
|
+
|
13
|
+
## Features
|
14
|
+
|
15
|
+
- **Multiple data sources** - PostgreSQL, MySQL, Redshift, and [many more](#full-list)
|
16
|
+
- **Variables** - run the same queries with different values
|
17
|
+
- **Checks & alerts** - get emailed when bad data appears
|
18
|
+
- **Audits** - all queries are tracked
|
19
|
+
- **Security** - works with your authentication system
|
20
|
+
|
21
|
+
## Docs
|
22
|
+
|
23
|
+
- [Installation](#installation)
|
24
|
+
- [Queries](#queries)
|
25
|
+
- [Charts](#charts)
|
26
|
+
- [Dashboards](#dashboards)
|
27
|
+
- [Checks](#checks)
|
28
|
+
|
29
|
+
## Installation
|
30
|
+
|
31
|
+
Add this line to your application’s Gemfile:
|
32
|
+
|
33
|
+
```ruby
|
34
|
+
gem 'blazer'
|
35
|
+
```
|
36
|
+
|
37
|
+
Run:
|
38
|
+
|
39
|
+
```sh
|
40
|
+
rails generate blazer:install
|
41
|
+
rails db:migrate
|
42
|
+
```
|
43
|
+
|
44
|
+
And mount the dashboard in your `config/routes.rb`:
|
45
|
+
|
46
|
+
```ruby
|
47
|
+
mount Blazer::Engine, at: "blazer"
|
48
|
+
```
|
49
|
+
|
50
|
+
For production, specify your database:
|
51
|
+
|
52
|
+
```ruby
|
53
|
+
ENV["BLAZER_DATABASE_URL"] = "postgres://user:password@hostname:5432/database"
|
54
|
+
```
|
55
|
+
|
56
|
+
Blazer tries to protect against queries which modify data (by running each query in a transaction and rolling it back), but a safer approach is to use a read only user. [See how to create one](#permissions).
|
57
|
+
|
58
|
+
#### Checks (optional)
|
59
|
+
|
60
|
+
Be sure to set a host in `config/environments/production.rb` for emails to work.
|
61
|
+
|
62
|
+
```ruby
|
63
|
+
config.action_mailer.default_url_options = {host: "blazer.dokkuapp.com"}
|
64
|
+
```
|
65
|
+
|
66
|
+
Schedule checks to run (with cron, [Heroku Scheduler](https://elements.heroku.com/addons/scheduler), etc). The default options are every 5 minutes, 1 hour, or 1 day, which you can customize. For each of these options, set up a task to run.
|
67
|
+
|
68
|
+
```sh
|
69
|
+
rake blazer:run_checks SCHEDULE="5 minutes"
|
70
|
+
rake blazer:run_checks SCHEDULE="1 hour"
|
71
|
+
rake blazer:run_checks SCHEDULE="1 day"
|
72
|
+
```
|
73
|
+
|
74
|
+
You can also set up failing checks to be sent once a day (or whatever you prefer).
|
75
|
+
|
76
|
+
```sh
|
77
|
+
rake blazer:send_failing_checks
|
78
|
+
```
|
79
|
+
|
80
|
+
Here’s what it looks like with cron.
|
81
|
+
|
82
|
+
```
|
83
|
+
*/5 * * * * rake blazer:run_checks SCHEDULE="5 minutes"
|
84
|
+
0 * * * * rake blazer:run_checks SCHEDULE="1 hour"
|
85
|
+
30 7 * * * rake blazer:run_checks SCHEDULE="1 day"
|
86
|
+
0 8 * * * rake blazer:send_failing_checks
|
87
|
+
```
|
88
|
+
|
89
|
+
For Slack notifications, create an [incoming webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) and set:
|
90
|
+
|
91
|
+
```sh
|
92
|
+
BLAZER_SLACK_WEBHOOK_URL=https://hooks.slack.com/...
|
93
|
+
```
|
94
|
+
|
95
|
+
Name the webhook “Blazer” and add a cool icon.
|
96
|
+
|
97
|
+
## Authentication
|
98
|
+
|
99
|
+
Don’t forget to protect the dashboard in production.
|
100
|
+
|
101
|
+
### Basic Authentication
|
102
|
+
|
103
|
+
Set the following variables in your environment or an initializer.
|
104
|
+
|
105
|
+
```ruby
|
106
|
+
ENV["BLAZER_USERNAME"] = "andrew"
|
107
|
+
ENV["BLAZER_PASSWORD"] = "secret"
|
108
|
+
```
|
109
|
+
|
110
|
+
### Devise
|
111
|
+
|
112
|
+
```ruby
|
113
|
+
authenticate :user, ->(user) { user.admin? } do
|
114
|
+
mount Blazer::Engine, at: "blazer"
|
115
|
+
end
|
116
|
+
```
|
117
|
+
|
118
|
+
### Other
|
119
|
+
|
120
|
+
Specify a `before_action` method to run in `blazer.yml`.
|
121
|
+
|
122
|
+
```yml
|
123
|
+
before_action_method: require_admin
|
124
|
+
```
|
125
|
+
|
126
|
+
You can define this method in your `ApplicationController`.
|
127
|
+
|
128
|
+
```ruby
|
129
|
+
def require_admin
|
130
|
+
# depending on your auth, something like...
|
131
|
+
redirect_to root_path unless current_user && current_user.admin?
|
132
|
+
end
|
133
|
+
```
|
134
|
+
|
135
|
+
Be sure to render or redirect for unauthorized users.
|
136
|
+
|
137
|
+
## Permissions
|
138
|
+
|
139
|
+
Blazer runs each query in a transaction and rolls it back to prevent queries from modifying data. As an additional line of defense, we recommend using a read only user.
|
140
|
+
|
141
|
+
### PostgreSQL
|
142
|
+
|
143
|
+
Create a user with read only permissions:
|
144
|
+
|
145
|
+
```sql
|
146
|
+
BEGIN;
|
147
|
+
CREATE ROLE blazer LOGIN PASSWORD 'secret123';
|
148
|
+
GRANT CONNECT ON DATABASE database_name TO blazer;
|
149
|
+
GRANT USAGE ON SCHEMA public TO blazer;
|
150
|
+
GRANT SELECT ON ALL TABLES IN SCHEMA public TO blazer;
|
151
|
+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO blazer;
|
152
|
+
COMMIT;
|
153
|
+
```
|
154
|
+
|
155
|
+
### MySQL
|
156
|
+
|
157
|
+
Create a user with read only permissions:
|
158
|
+
|
159
|
+
```sql
|
160
|
+
GRANT SELECT, SHOW VIEW ON database_name.* TO blazer@’127.0.0.1′ IDENTIFIED BY ‘secret123‘;
|
161
|
+
FLUSH PRIVILEGES;
|
162
|
+
```
|
163
|
+
|
164
|
+
### MongoDB
|
165
|
+
|
166
|
+
Create a user with read only permissions:
|
167
|
+
|
168
|
+
```
|
169
|
+
db.createUser({user: "blazer", pwd: "password", roles: ["read"]})
|
170
|
+
```
|
171
|
+
|
172
|
+
Also, make sure authorization is enabled when you start the server.
|
173
|
+
|
174
|
+
## Sensitive Data
|
175
|
+
|
176
|
+
If your database contains sensitive or personal data, check out [Hypershield](https://github.com/ankane/hypershield) to shield it.
|
177
|
+
|
178
|
+
## Encrypted Data
|
179
|
+
|
180
|
+
If you need to search encrypted data, use [blind indexing](https://github.com/ankane/blind_index).
|
181
|
+
|
182
|
+
You can have Blazer transform specific variables with:
|
183
|
+
|
184
|
+
```ruby
|
185
|
+
Blazer.transform_variable = lambda do |name, value|
|
186
|
+
value = User.compute_email_bidx(value) if name == "email_bidx"
|
187
|
+
value
|
188
|
+
end
|
189
|
+
```
|
190
|
+
|
191
|
+
## Queries
|
192
|
+
|
193
|
+
### Variables
|
194
|
+
|
195
|
+
Create queries with variables.
|
196
|
+
|
197
|
+
```sql
|
198
|
+
SELECT * FROM users WHERE gender = {gender}
|
199
|
+
```
|
200
|
+
|
201
|
+
Use `{start_time}` and `{end_time}` for time ranges. [Example](https://blazer.dokkuapp.com/queries/9-time-range-selector?start_time=1997-10-03T05%3A00%3A00%2B00%3A00&end_time=1997-10-04T04%3A59%3A59%2B00%3A00)
|
202
|
+
|
203
|
+
```sql
|
204
|
+
SELECT * FROM ratings WHERE rated_at >= {start_time} AND rated_at <= {end_time}
|
205
|
+
```
|
206
|
+
|
207
|
+
### Smart Variables
|
208
|
+
|
209
|
+
[Example](https://blazer.dokkuapp.com/queries/1-smart-variable)
|
210
|
+
|
211
|
+
Suppose you have the query:
|
212
|
+
|
213
|
+
```sql
|
214
|
+
SELECT * FROM users WHERE occupation_id = {occupation_id}
|
215
|
+
```
|
216
|
+
|
217
|
+
Instead of remembering each occupation’s id, users can select occupations by name.
|
218
|
+
|
219
|
+
Add a smart variable with:
|
220
|
+
|
221
|
+
```yml
|
222
|
+
smart_variables:
|
223
|
+
occupation_id: "SELECT id, name FROM occupations ORDER BY name ASC"
|
224
|
+
```
|
225
|
+
|
226
|
+
The first column is the value of the variable, and the second column is the label.
|
227
|
+
|
228
|
+
You can also use an array or hash for static data and enums.
|
229
|
+
|
230
|
+
```yml
|
231
|
+
smart_variables:
|
232
|
+
period: ["day", "week", "month"]
|
233
|
+
status: {0: "Active", 1: "Archived"}
|
234
|
+
```
|
235
|
+
|
236
|
+
### Linked Columns
|
237
|
+
|
238
|
+
[Example](https://blazer.dokkuapp.com/queries/3-linked-column) - title column
|
239
|
+
|
240
|
+
Link results to other pages in your apps or around the web. Specify a column name and where it should link to. You can use the value of the result with `{value}`.
|
241
|
+
|
242
|
+
```yml
|
243
|
+
linked_columns:
|
244
|
+
user_id: "/admin/users/{value}"
|
245
|
+
ip_address: "https://www.infosniper.net/index.php?ip_address={value}"
|
246
|
+
```
|
247
|
+
|
248
|
+
### Smart Columns
|
249
|
+
|
250
|
+
[Example](https://blazer.dokkuapp.com/queries/2-smart-column) - occupation_id column
|
251
|
+
|
252
|
+
Suppose you have the query:
|
253
|
+
|
254
|
+
```sql
|
255
|
+
SELECT name, city_id FROM users
|
256
|
+
```
|
257
|
+
|
258
|
+
See which city the user belongs to without a join.
|
259
|
+
|
260
|
+
```yml
|
261
|
+
smart_columns:
|
262
|
+
city_id: "SELECT id, name FROM cities WHERE id IN {value}"
|
263
|
+
```
|
264
|
+
|
265
|
+
You can also use a hash for static data and enums.
|
266
|
+
|
267
|
+
```yml
|
268
|
+
smart_columns:
|
269
|
+
status: {0: "Active", 1: "Archived"}
|
270
|
+
```
|
271
|
+
|
272
|
+
### Caching
|
273
|
+
|
274
|
+
Blazer can automatically cache results to improve speed. It can cache slow queries:
|
275
|
+
|
276
|
+
```yml
|
277
|
+
cache:
|
278
|
+
mode: slow
|
279
|
+
expires_in: 60 # min
|
280
|
+
slow_threshold: 15 # sec
|
281
|
+
```
|
282
|
+
|
283
|
+
Or it can cache all queries:
|
284
|
+
|
285
|
+
```yml
|
286
|
+
cache:
|
287
|
+
mode: all
|
288
|
+
expires_in: 60 # min
|
289
|
+
```
|
290
|
+
|
291
|
+
Of course, you can force a refresh at any time.
|
292
|
+
|
293
|
+
## Charts
|
294
|
+
|
295
|
+
Blazer will automatically generate charts based on the types of the columns returned in your query.
|
296
|
+
|
297
|
+
**Note:** The order of columns matters.
|
298
|
+
|
299
|
+
### Line Chart
|
300
|
+
|
301
|
+
There are two ways to generate line charts.
|
302
|
+
|
303
|
+
2+ columns - timestamp, numeric(s) - [Example](https://blazer.dokkuapp.com/queries/4-line-chart-format-1)
|
304
|
+
|
305
|
+
```sql
|
306
|
+
SELECT date_trunc('week', created_at), COUNT(*) FROM users GROUP BY 1
|
307
|
+
```
|
308
|
+
|
309
|
+
3 columns - timestamp, string, numeric - [Example](https://blazer.dokkuapp.com/queries/5-line-chart-format-2)
|
310
|
+
|
311
|
+
|
312
|
+
```sql
|
313
|
+
SELECT date_trunc('week', created_at), gender, COUNT(*) FROM users GROUP BY 1, 2
|
314
|
+
```
|
315
|
+
|
316
|
+
### Column Chart
|
317
|
+
|
318
|
+
There are also two ways to generate column charts.
|
319
|
+
|
320
|
+
2+ columns - string, numeric(s) - [Example](https://blazer.dokkuapp.com/queries/6-column-chart-format-1)
|
321
|
+
|
322
|
+
```sql
|
323
|
+
SELECT gender, COUNT(*) FROM users GROUP BY 1
|
324
|
+
```
|
325
|
+
|
326
|
+
3 columns - string, string, numeric - [Example](https://blazer.dokkuapp.com/queries/7-column-chart-format-2)
|
327
|
+
|
328
|
+
```sql
|
329
|
+
SELECT gender, zip_code, COUNT(*) FROM users GROUP BY 1, 2
|
330
|
+
```
|
331
|
+
|
332
|
+
### Scatter Chart
|
333
|
+
|
334
|
+
2 columns - both numeric - [Example](https://blazer.dokkuapp.com/queries/16-scatter-chart)
|
335
|
+
|
336
|
+
```sql
|
337
|
+
SELECT x, y FROM table
|
338
|
+
```
|
339
|
+
|
340
|
+
### Pie Chart
|
341
|
+
|
342
|
+
2 columns - string, numeric - and last column named `pie` - [Example](https://blazer.dokkuapp.com/queries/17-pie-chart)
|
343
|
+
|
344
|
+
```sql
|
345
|
+
SELECT gender, COUNT(*) AS pie FROM users GROUP BY 1
|
346
|
+
```
|
347
|
+
|
348
|
+
### Maps
|
349
|
+
|
350
|
+
Columns named `latitude` and `longitude` or `lat` and `lon` or `lat` and `lng` - [Example](https://blazer.dokkuapp.com/queries/15-map)
|
351
|
+
|
352
|
+
```sql
|
353
|
+
SELECT name, latitude, longitude FROM cities
|
354
|
+
```
|
355
|
+
|
356
|
+
To enable, get an access token from [Mapbox](https://www.mapbox.com/) and set `ENV["MAPBOX_ACCESS_TOKEN"]`.
|
357
|
+
|
358
|
+
### Targets
|
359
|
+
|
360
|
+
Use the column name `target` to draw a line for goals. [Example](https://blazer.dokkuapp.com/queries/8-target-line)
|
361
|
+
|
362
|
+
```sql
|
363
|
+
SELECT date_trunc('week', created_at), COUNT(*) AS new_users, 100000 AS target FROM users GROUP BY 1
|
364
|
+
```
|
365
|
+
|
366
|
+
## Dashboards
|
367
|
+
|
368
|
+
Create a dashboard with multiple queries. [Example](https://blazer.dokkuapp.com/dashboards/1-dashboard-demo)
|
369
|
+
|
370
|
+
If the query has a chart, the chart is shown. Otherwise, you’ll see a table.
|
371
|
+
|
372
|
+
If any queries have variables, they will show up on the dashboard.
|
373
|
+
|
374
|
+
## Checks
|
375
|
+
|
376
|
+
Checks give you a centralized place to see the health of your data. [Example](https://blazer.dokkuapp.com/checks)
|
377
|
+
|
378
|
+
Create a query to identify bad rows.
|
379
|
+
|
380
|
+
```sql
|
381
|
+
SELECT * FROM ratings WHERE user_id IS NULL /* all ratings should have a user */
|
382
|
+
```
|
383
|
+
|
384
|
+
Then create check with optional emails if you want to be notified. Emails are sent when a check starts failing, and when it starts passing again.
|
385
|
+
|
386
|
+
## Anomaly Detection
|
387
|
+
|
388
|
+
Blazer supports two different approaches to anomaly detection.
|
389
|
+
|
390
|
+
### Trend
|
391
|
+
|
392
|
+
[Trend](https://trendapi.org/) is easiest to set up but uses an external service.
|
393
|
+
|
394
|
+
Add [trend](https://github.com/ankane/trend) to your Gemfile:
|
395
|
+
|
396
|
+
```ruby
|
397
|
+
gem 'trend'
|
398
|
+
```
|
399
|
+
|
400
|
+
And add to `config/blazer.yml`:
|
401
|
+
|
402
|
+
```yml
|
403
|
+
anomaly_checks: trend
|
404
|
+
```
|
405
|
+
|
406
|
+
### R
|
407
|
+
|
408
|
+
R is harder to set up but doesn’t use an external service. It uses Twitter’s [AnomalyDetection](https://github.com/twitter/AnomalyDetection) library.
|
409
|
+
|
410
|
+
First, [install R](https://cloud.r-project.org/). Then, run:
|
411
|
+
|
412
|
+
```R
|
413
|
+
install.packages("remotes")
|
414
|
+
remotes::install_github("twitter/AnomalyDetection")
|
415
|
+
```
|
416
|
+
|
417
|
+
And add to `config/blazer.yml`:
|
418
|
+
|
419
|
+
```yml
|
420
|
+
anomaly_checks: r
|
421
|
+
```
|
422
|
+
|
423
|
+
If upgrading from version 1.4 or below, also follow the [upgrade instructions](#15).
|
424
|
+
|
425
|
+
If you’re on Heroku, follow [these additional instructions](#anomaly-detection-on-heroku).
|
426
|
+
|
427
|
+
## Forecasting
|
428
|
+
|
429
|
+
Blazer has experimental support for two different forecasting methods.
|
430
|
+
|
431
|
+
A forecast link will appear for queries that return 2 columns with types timestamp and numeric.
|
432
|
+
|
433
|
+
[Example](https://blazer.dokkuapp.com/queries/18-forecast?forecast=t)
|
434
|
+
|
435
|
+
### Prophet
|
436
|
+
|
437
|
+
Add [prophet](https://github.com/ankane/prophet) to your Gemfile:
|
438
|
+
|
439
|
+
```ruby
|
440
|
+
gem 'prophet-rb', '>= 0.2.1'
|
441
|
+
```
|
442
|
+
|
443
|
+
And add to `config/blazer.yml`:
|
444
|
+
|
445
|
+
```yml
|
446
|
+
forecasting: prophet
|
447
|
+
```
|
448
|
+
|
449
|
+
### Trend
|
450
|
+
|
451
|
+
[Trend](https://trendapi.org/) uses an external service.
|
452
|
+
|
453
|
+
Add [trend](https://github.com/ankane/trend) to your Gemfile:
|
454
|
+
|
455
|
+
```ruby
|
456
|
+
gem 'trend'
|
457
|
+
```
|
458
|
+
|
459
|
+
And add to `config/blazer.yml`:
|
460
|
+
|
461
|
+
```yml
|
462
|
+
forecasting: trend
|
463
|
+
```
|
464
|
+
|
465
|
+
## Data Sources
|
466
|
+
|
467
|
+
Blazer supports multiple data sources :tada:
|
468
|
+
|
469
|
+
Add additional data sources in `config/blazer.yml`:
|
470
|
+
|
471
|
+
```yml
|
472
|
+
data_sources:
|
473
|
+
main:
|
474
|
+
url: <%= ENV["BLAZER_DATABASE_URL"] %>
|
475
|
+
# timeout, smart_variables, linked_columns, smart_columns
|
476
|
+
catalog:
|
477
|
+
url: <%= ENV["CATALOG_DATABASE_URL"] %>
|
478
|
+
# ...
|
479
|
+
redshift:
|
480
|
+
url: <%= ENV["REDSHIFT_DATABASE_URL"] %>
|
481
|
+
# ...
|
482
|
+
```
|
483
|
+
|
484
|
+
### Full List
|
485
|
+
|
486
|
+
- [Amazon Athena](#amazon-athena)
|
487
|
+
- [Amazon Redshift](#amazon-redshift)
|
488
|
+
- [Apache Drill](#apache-drill)
|
489
|
+
- [Cassandra](#cassandra)
|
490
|
+
- [Druid](#druid)
|
491
|
+
- [Elasticsearch](#elasticsearch)
|
492
|
+
- [Google BigQuery](#google-bigquery)
|
493
|
+
- [IBM DB2 and Informix](#ibm-db2-and-informix)
|
494
|
+
- [InfluxDB](#influxdb)
|
495
|
+
- [MongoDB](#mongodb-1)
|
496
|
+
- [MySQL](#mysql-1)
|
497
|
+
- [Neo4j](#neo4j)
|
498
|
+
- [Oracle](#oracle)
|
499
|
+
- [PostgreSQL](#postgresql-1)
|
500
|
+
- [Presto](#presto)
|
501
|
+
- [Salesforce](#salesforce)
|
502
|
+
- [Socrata Open Data API (SODA)](#socrata-open-data-api-soda)
|
503
|
+
- [Snowflake](#snowflake)
|
504
|
+
- [SQLite](#sqlite)
|
505
|
+
- [SQL Server](#sql-server)
|
506
|
+
|
507
|
+
You can also [create an adapter](#creating-an-adapter) for any other data store.
|
508
|
+
|
509
|
+
**Note:** In the examples below, we recommend using environment variables for urls.
|
510
|
+
|
511
|
+
```yml
|
512
|
+
data_sources:
|
513
|
+
my_source:
|
514
|
+
url: <%= ENV["BLAZER_MY_SOURCE_URL"] %>
|
515
|
+
```
|
516
|
+
|
517
|
+
### Amazon Athena
|
518
|
+
|
519
|
+
Add [aws-sdk-athena](https://github.com/aws/aws-sdk-ruby) and [aws-sdk-glue](https://github.com/aws/aws-sdk-ruby) to your Gemfile and set:
|
520
|
+
|
521
|
+
```yml
|
522
|
+
data_sources:
|
523
|
+
my_source:
|
524
|
+
adapter: athena
|
525
|
+
database: database
|
526
|
+
output_location: s3://some-bucket/
|
527
|
+
```
|
528
|
+
|
529
|
+
### Amazon Redshift
|
530
|
+
|
531
|
+
Add [activerecord6-redshift-adapter](https://github.com/kwent/activerecord6-redshift-adapter) or [activerecord5-redshift-adapter](https://github.com/ConsultingMD/activerecord5-redshift-adapter) to your Gemfile and set:
|
532
|
+
|
533
|
+
```yml
|
534
|
+
data_sources:
|
535
|
+
my_source:
|
536
|
+
url: redshift://user:password@hostname:5439/database
|
537
|
+
```
|
538
|
+
|
539
|
+
### Apache Drill
|
540
|
+
|
541
|
+
Add [drill-sergeant](https://github.com/ankane/drill-sergeant) to your Gemfile and set:
|
542
|
+
|
543
|
+
```yml
|
544
|
+
data_sources:
|
545
|
+
my_source:
|
546
|
+
adapter: drill
|
547
|
+
url: http://hostname:8047
|
548
|
+
```
|
549
|
+
|
550
|
+
### Cassandra
|
551
|
+
|
552
|
+
Add [cassandra-driver](https://github.com/datastax/ruby-driver) to your Gemfile and set:
|
553
|
+
|
554
|
+
```yml
|
555
|
+
data_sources:
|
556
|
+
my_source:
|
557
|
+
url: cassandra://user:password@hostname:9042/keyspace
|
558
|
+
```
|
559
|
+
|
560
|
+
### Druid
|
561
|
+
|
562
|
+
Enable [SQL support](http://druid.io/docs/latest/querying/sql.html#configuration) on the broker and set:
|
563
|
+
|
564
|
+
```yml
|
565
|
+
data_sources:
|
566
|
+
my_source:
|
567
|
+
adapter: druid
|
568
|
+
url: http://hostname:8082
|
569
|
+
```
|
570
|
+
|
571
|
+
### Elasticsearch
|
572
|
+
|
573
|
+
Add [elasticsearch](https://github.com/elastic/elasticsearch-ruby) and [elasticsearch-xpack](https://github.com/elastic/elasticsearch-ruby/tree/master/elasticsearch-xpack) to your Gemfile and set:
|
574
|
+
|
575
|
+
```yml
|
576
|
+
data_sources:
|
577
|
+
my_source:
|
578
|
+
adapter: elasticsearch
|
579
|
+
url: http://user:password@hostname:9200
|
580
|
+
```
|
581
|
+
|
582
|
+
### Google BigQuery
|
583
|
+
|
584
|
+
Add [google-cloud-bigquery](https://github.com/GoogleCloudPlatform/google-cloud-ruby/tree/master/google-cloud-bigquery) to your Gemfile and set:
|
585
|
+
|
586
|
+
```yml
|
587
|
+
data_sources:
|
588
|
+
my_source:
|
589
|
+
adapter: bigquery
|
590
|
+
project: your-project
|
591
|
+
keyfile: path/to/keyfile.json
|
592
|
+
```
|
593
|
+
|
594
|
+
### IBM DB2 and Informix
|
595
|
+
|
596
|
+
Add [ibm_db](https://github.com/ibmdb/ruby-ibmdb) to your Gemfile and set:
|
597
|
+
|
598
|
+
```yml
|
599
|
+
data_sources:
|
600
|
+
my_source:
|
601
|
+
url: ibm-db://user:password@hostname:50000/database
|
602
|
+
```
|
603
|
+
|
604
|
+
### InfluxDB
|
605
|
+
|
606
|
+
*Experimental*
|
607
|
+
|
608
|
+
Add [influxdb](https://github.com/influxdata/influxdb-ruby) to your Gemfile and set:
|
609
|
+
|
610
|
+
```yml
|
611
|
+
data_sources:
|
612
|
+
my_source:
|
613
|
+
adapter: influxdb
|
614
|
+
url: http://user:password@hostname:8086/database
|
615
|
+
```
|
616
|
+
|
617
|
+
Supports [InfluxQL](https://docs.influxdata.com/influxdb/v1.8/query_language/explore-data/)
|
618
|
+
|
619
|
+
### MongoDB
|
620
|
+
|
621
|
+
Add [mongo](https://github.com/mongodb/mongo-ruby-driver) to your Gemfile and set:
|
622
|
+
|
623
|
+
```yml
|
624
|
+
data_sources:
|
625
|
+
my_source:
|
626
|
+
url: mongodb://user:password@hostname:27017/database
|
627
|
+
```
|
628
|
+
|
629
|
+
### MySQL
|
630
|
+
|
631
|
+
Add [mysql2](https://github.com/brianmario/mysql2) to your Gemfile (if it’s not there) and set:
|
632
|
+
|
633
|
+
```yml
|
634
|
+
data_sources:
|
635
|
+
my_source:
|
636
|
+
url: mysql2://user:password@hostname:3306/database
|
637
|
+
```
|
638
|
+
|
639
|
+
### Neo4j
|
640
|
+
|
641
|
+
*Experimental*
|
642
|
+
|
643
|
+
Add [neo4j-core](https://github.com/neo4jrb/neo4j-core) to your Gemfile and set:
|
644
|
+
|
645
|
+
```yml
|
646
|
+
data_sources:
|
647
|
+
my_source:
|
648
|
+
adapter: neo4j
|
649
|
+
url: http://user:password@hostname:7474
|
650
|
+
```
|
651
|
+
|
652
|
+
### Oracle
|
653
|
+
|
654
|
+
Add [activerecord-oracle_enhanced-adapter](https://github.com/rsim/oracle-enhanced) and [ruby-oci8](https://github.com/kubo/ruby-oci8) to your Gemfile and set:
|
655
|
+
|
656
|
+
```yml
|
657
|
+
data_sources:
|
658
|
+
my_source:
|
659
|
+
url: oracle-enhanced://user:password@hostname:1521/database
|
660
|
+
```
|
661
|
+
|
662
|
+
### PostgreSQL
|
663
|
+
|
664
|
+
Add [pg](https://github.com/ged/ruby-pg) to your Gemfile (if it’s not there) and set:
|
665
|
+
|
666
|
+
```yml
|
667
|
+
data_sources:
|
668
|
+
my_source:
|
669
|
+
url: postgres://user:password@hostname:5432/database
|
670
|
+
```
|
671
|
+
|
672
|
+
### Presto
|
673
|
+
|
674
|
+
Add [presto-client](https://github.com/treasure-data/presto-client-ruby) to your Gemfile and set:
|
675
|
+
|
676
|
+
```yml
|
677
|
+
data_sources:
|
678
|
+
my_source:
|
679
|
+
url: presto://user@hostname:8080/catalog
|
680
|
+
```
|
681
|
+
|
682
|
+
### Salesforce
|
683
|
+
|
684
|
+
*Experimental*
|
685
|
+
|
686
|
+
Add [restforce](https://github.com/restforce/restforce) to your Gemfile and set:
|
687
|
+
|
688
|
+
```yml
|
689
|
+
data_sources:
|
690
|
+
my_source:
|
691
|
+
adapter: salesforce
|
692
|
+
```
|
693
|
+
|
694
|
+
And set the appropriate environment variables:
|
695
|
+
|
696
|
+
```sh
|
697
|
+
SALESFORCE_USERNAME="username"
|
698
|
+
SALESFORCE_PASSWORD="password"
|
699
|
+
SALESFORCE_SECURITY_TOKEN="security token"
|
700
|
+
SALESFORCE_CLIENT_ID="client id"
|
701
|
+
SALESFORCE_CLIENT_SECRET="client secret"
|
702
|
+
SALESFORCE_API_VERSION="41.0"
|
703
|
+
```
|
704
|
+
|
705
|
+
Supports [SOQL](https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql.htm)
|
706
|
+
|
707
|
+
### Socrata Open Data API (SODA)
|
708
|
+
|
709
|
+
*Experimental*
|
710
|
+
|
711
|
+
Set:
|
712
|
+
|
713
|
+
```yml
|
714
|
+
data_sources:
|
715
|
+
my_source:
|
716
|
+
adapter: soda
|
717
|
+
url: https://soda.demo.socrata.com/resource/4tka-6guv.json
|
718
|
+
app_token: ...
|
719
|
+
```
|
720
|
+
|
721
|
+
Supports [SoQL](https://dev.socrata.com/docs/functions/)
|
722
|
+
|
723
|
+
### Snowflake
|
724
|
+
|
725
|
+
First, install ODBC. For Homebrew, use:
|
726
|
+
|
727
|
+
```sh
|
728
|
+
brew install unixodbc
|
729
|
+
```
|
730
|
+
|
731
|
+
For Ubuntu, use:
|
732
|
+
|
733
|
+
```sh
|
734
|
+
sudo apt-get install unixodbc-dev
|
735
|
+
```
|
736
|
+
|
737
|
+
For Heroku, use the [Apt buildpack](https://github.com/heroku/heroku-buildpack-apt) and create an `Aptfile` with:
|
738
|
+
|
739
|
+
```text
|
740
|
+
unixodbc-dev
|
741
|
+
https://sfc-repo.snowflakecomputing.com/odbc/linux/2.21.5/snowflake-odbc-2.21.5.x86_64.deb
|
742
|
+
```
|
743
|
+
|
744
|
+
> This installs the driver at `/app/.apt/usr/lib/snowflake/odbc/lib/libSnowflake.so`
|
745
|
+
|
746
|
+
Then, download the [Snowflake ODBC driver](https://docs.snowflake.net/manuals/user-guide/odbc-download.html). Add [odbc_adapter](https://github.com/localytics/odbc_adapter) to your Gemfile and set:
|
747
|
+
|
748
|
+
```yml
|
749
|
+
data_sources:
|
750
|
+
my_source:
|
751
|
+
adapter: snowflake
|
752
|
+
conn_str: Driver=/path/to/libSnowflake.so;uid=user;pwd=password;server=host.snowflakecomputing.com
|
753
|
+
```
|
754
|
+
|
755
|
+
### SQLite
|
756
|
+
|
757
|
+
Add [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) to your Gemfile and set:
|
758
|
+
|
759
|
+
```yml
|
760
|
+
data_sources:
|
761
|
+
my_source:
|
762
|
+
url: sqlite3:path/to/database.sqlite3
|
763
|
+
```
|
764
|
+
|
765
|
+
### SQL Server
|
766
|
+
|
767
|
+
Add [tiny_tds](https://github.com/rails-sqlserver/tiny_tds) and [activerecord-sqlserver-adapter](https://github.com/rails-sqlserver/activerecord-sqlserver-adapter) to your Gemfile and set:
|
768
|
+
|
769
|
+
```yml
|
770
|
+
data_sources:
|
771
|
+
my_source:
|
772
|
+
url: sqlserver://user:password@hostname:1433/database
|
773
|
+
```
|
774
|
+
|
775
|
+
## Creating an Adapter
|
776
|
+
|
777
|
+
Create an adapter for any data store with:
|
778
|
+
|
779
|
+
```ruby
|
780
|
+
class FooAdapter < Blazer::Adapters::BaseAdapter
|
781
|
+
# code goes here
|
782
|
+
end
|
783
|
+
|
784
|
+
Blazer.register_adapter "foo", FooAdapter
|
785
|
+
```
|
786
|
+
|
787
|
+
See the [Presto adapter](https://github.com/ankane/blazer/blob/master/lib/blazer/adapters/presto_adapter.rb) for a good example. Then use:
|
788
|
+
|
789
|
+
```yml
|
790
|
+
data_sources:
|
791
|
+
my_source:
|
792
|
+
adapter: foo
|
793
|
+
url: http://user:password@hostname:9200/
|
794
|
+
```
|
795
|
+
|
796
|
+
## Query Permissions
|
797
|
+
|
798
|
+
Blazer supports a basic permissions model.
|
799
|
+
|
800
|
+
1. Queries without a name are unlisted
|
801
|
+
2. Queries whose name starts with `#` are only listed to the creator
|
802
|
+
3. Queries whose name starts with `*` can only be edited by the creator
|
803
|
+
|
804
|
+
## Learn SQL
|
805
|
+
|
806
|
+
Have team members who want to learn SQL? Here are a few great, free resources.
|
807
|
+
|
808
|
+
- [The Data School](https://dataschool.com/learn-sql/)
|
809
|
+
- [SQLBolt](https://sqlbolt.com/)
|
810
|
+
|
811
|
+
## Useful Tools
|
812
|
+
|
813
|
+
For an easy way to group by day, week, month, and more with correct time zones, check out [Groupdate.sql](https://github.com/ankane/groupdate.sql).
|
814
|
+
|
815
|
+
## Standalone Version
|
816
|
+
|
817
|
+
Looking for a standalone version? Check out [Ghost Blazer](https://github.com/buren/ghost_blazer).
|
818
|
+
|
819
|
+
## Performance
|
820
|
+
|
821
|
+
By default, queries take up a request while they are running. To run queries asynchronously, add to your config:
|
822
|
+
|
823
|
+
```yml
|
824
|
+
async: true
|
825
|
+
```
|
826
|
+
|
827
|
+
**Note:** Requires Rails 5+ and caching to be enabled. If you have multiple web processes, your app must use a centralized cache store like Memcached or Redis.
|
828
|
+
|
829
|
+
```ruby
|
830
|
+
config.cache_store = :mem_cache_store
|
831
|
+
```
|
832
|
+
|
833
|
+
## Anomaly Detection on Heroku
|
834
|
+
|
835
|
+
Add the [R buildpack](https://github.com/virtualstaticvoid/heroku-buildpack-r) to your app.
|
836
|
+
|
837
|
+
```sh
|
838
|
+
heroku buildpacks:add --index 1 https://github.com/virtualstaticvoid/heroku-buildpack-r.git\#cedar-14
|
839
|
+
```
|
840
|
+
|
841
|
+
And create an `init.r` with:
|
842
|
+
|
843
|
+
```sh
|
844
|
+
if (!"AnomalyDetection" %in% installed.packages()) {
|
845
|
+
install.packages("devtools")
|
846
|
+
devtools::install_github("twitter/AnomalyDetection")
|
847
|
+
}
|
848
|
+
```
|
849
|
+
|
850
|
+
Commit and deploy away. The first deploy may take a few minutes.
|
851
|
+
|
852
|
+
## Content Security Policy
|
853
|
+
|
854
|
+
If views are stuck with a `Loading...` message, there might be a problem with strict CSP settings in your app. This can be checked with Firefox or Chrome dev tools. You can allow Blazer to override these settings for its controllers with:
|
855
|
+
|
856
|
+
```yml
|
857
|
+
override_csp: true
|
858
|
+
```
|
859
|
+
|
860
|
+
## Upgrading
|
861
|
+
|
862
|
+
### 2.0
|
863
|
+
|
864
|
+
To use Slack notifications, create a migration
|
865
|
+
|
866
|
+
```sh
|
867
|
+
rails g migration add_slack_channels_to_blazer_checks
|
868
|
+
```
|
869
|
+
|
870
|
+
with:
|
871
|
+
|
872
|
+
```ruby
|
873
|
+
add_column :blazer_checks, :slack_channels, :text
|
874
|
+
```
|
875
|
+
|
876
|
+
### 1.5
|
877
|
+
|
878
|
+
To take advantage of the anomaly detection, create a migration
|
879
|
+
|
880
|
+
```sh
|
881
|
+
rails g migration upgrade_blazer_to_1_5
|
882
|
+
```
|
883
|
+
|
884
|
+
with:
|
885
|
+
|
886
|
+
```ruby
|
887
|
+
add_column :blazer_checks, :check_type, :string
|
888
|
+
add_column :blazer_checks, :message, :text
|
889
|
+
commit_db_transaction
|
890
|
+
|
891
|
+
Blazer::Check.reset_column_information
|
892
|
+
|
893
|
+
Blazer::Check.where(invert: true).update_all(check_type: "missing_data")
|
894
|
+
Blazer::Check.where(check_type: nil).update_all(check_type: "bad_data")
|
895
|
+
```
|
896
|
+
|
897
|
+
### 1.3
|
898
|
+
|
899
|
+
To take advantage of the latest features, create a migration
|
900
|
+
|
901
|
+
```sh
|
902
|
+
rails g migration upgrade_blazer_to_1_3
|
903
|
+
```
|
904
|
+
|
905
|
+
with:
|
906
|
+
|
907
|
+
```ruby
|
908
|
+
add_column :blazer_dashboards, :creator_id, :integer
|
909
|
+
add_column :blazer_checks, :creator_id, :integer
|
910
|
+
add_column :blazer_checks, :invert, :boolean
|
911
|
+
add_column :blazer_checks, :schedule, :string
|
912
|
+
add_column :blazer_checks, :last_run_at, :timestamp
|
913
|
+
commit_db_transaction
|
914
|
+
|
915
|
+
Blazer::Check.update_all schedule: "1 hour"
|
916
|
+
```
|
917
|
+
|
918
|
+
### 1.0
|
919
|
+
|
920
|
+
Blazer 1.0 brings a number of new features:
|
921
|
+
|
922
|
+
- multiple data sources, including Redshift
|
923
|
+
- dashboards
|
924
|
+
- checks
|
925
|
+
|
926
|
+
To upgrade, run:
|
927
|
+
|
928
|
+
```sh
|
929
|
+
bundle update blazer
|
930
|
+
```
|
931
|
+
|
932
|
+
Create a migration
|
933
|
+
|
934
|
+
```sh
|
935
|
+
rails g migration upgrade_blazer_to_1_0
|
936
|
+
```
|
937
|
+
|
938
|
+
with:
|
939
|
+
|
940
|
+
```ruby
|
941
|
+
add_column :blazer_queries, :data_source, :string
|
942
|
+
add_column :blazer_audits, :data_source, :string
|
943
|
+
|
944
|
+
create_table :blazer_dashboards do |t|
|
945
|
+
t.text :name
|
946
|
+
t.timestamps
|
947
|
+
end
|
948
|
+
|
949
|
+
create_table :blazer_dashboard_queries do |t|
|
950
|
+
t.references :dashboard
|
951
|
+
t.references :query
|
952
|
+
t.integer :position
|
953
|
+
t.timestamps
|
954
|
+
end
|
955
|
+
|
956
|
+
create_table :blazer_checks do |t|
|
957
|
+
t.references :query
|
958
|
+
t.string :state
|
959
|
+
t.text :emails
|
960
|
+
t.timestamps
|
961
|
+
end
|
962
|
+
```
|
963
|
+
|
964
|
+
And run:
|
965
|
+
|
966
|
+
```sh
|
967
|
+
rails db:migrate
|
968
|
+
```
|
969
|
+
|
970
|
+
Update `config/blazer.yml` with:
|
971
|
+
|
972
|
+
```yml
|
973
|
+
# see https://github.com/ankane/blazer for more info
|
974
|
+
|
975
|
+
data_sources:
|
976
|
+
main:
|
977
|
+
url: <%= ENV["BLAZER_DATABASE_URL"] %>
|
978
|
+
|
979
|
+
# statement timeout, in seconds
|
980
|
+
# applies to PostgreSQL only
|
981
|
+
# none by default
|
982
|
+
# timeout: 15
|
983
|
+
|
984
|
+
# time to cache results, in minutes
|
985
|
+
# can greatly improve speed
|
986
|
+
# none by default
|
987
|
+
# cache: 60
|
988
|
+
|
989
|
+
# wrap queries in a transaction for safety
|
990
|
+
# not necessary if you use a read-only user
|
991
|
+
# true by default
|
992
|
+
# use_transaction: false
|
993
|
+
|
994
|
+
smart_variables:
|
995
|
+
# zone_id: "SELECT id, name FROM zones ORDER BY name ASC"
|
996
|
+
|
997
|
+
linked_columns:
|
998
|
+
# user_id: "/admin/users/{value}"
|
999
|
+
|
1000
|
+
smart_columns:
|
1001
|
+
# user_id: "SELECT id, name FROM users WHERE id IN {value}"
|
1002
|
+
|
1003
|
+
# create audits
|
1004
|
+
audit: true
|
1005
|
+
|
1006
|
+
# change the time zone
|
1007
|
+
# time_zone: "Pacific Time (US & Canada)"
|
1008
|
+
|
1009
|
+
# class name of the user model
|
1010
|
+
# user_class: User
|
1011
|
+
|
1012
|
+
# method name for the current user
|
1013
|
+
# user_method: current_user
|
1014
|
+
|
1015
|
+
# method name for the display name
|
1016
|
+
# user_name: name
|
1017
|
+
|
1018
|
+
# email to send checks from
|
1019
|
+
# from_email: blazer@example.org
|
1020
|
+
```
|
1021
|
+
|
1022
|
+
## History
|
1023
|
+
|
1024
|
+
View the [changelog](https://github.com/ankane/blazer/blob/master/CHANGELOG.md)
|
1025
|
+
|
1026
|
+
## Thanks
|
1027
|
+
|
1028
|
+
Blazer uses a number of awesome open source projects, including [Rails](https://github.com/rails/rails/), [Vue.js](https://github.com/vuejs/vue), [jQuery](https://github.com/jquery/jquery), [Bootstrap](https://github.com/twbs/bootstrap), [Selectize](https://github.com/brianreavis/selectize.js), [StickyTableHeaders](https://github.com/jmosbech/StickyTableHeaders), [Stupid jQuery Table Sort](https://github.com/joequery/Stupid-Table-Plugin), and [Date Range Picker](https://github.com/dangrossman/bootstrap-daterangepicker).
|
1029
|
+
|
1030
|
+
Demo data from [MovieLens](https://grouplens.org/datasets/movielens/).
|
1031
|
+
|
1032
|
+
## Want to Make Blazer Better?
|
1033
|
+
|
1034
|
+
That’s awesome! Here are a few ways you can help:
|
1035
|
+
|
1036
|
+
- [Report bugs](https://github.com/ankane/blazer/issues)
|
1037
|
+
- Fix bugs and [submit pull requests](https://github.com/ankane/blazer/pulls)
|
1038
|
+
- Write, clarify, or fix documentation
|
1039
|
+
- Suggest or add new features
|
1040
|
+
|
1041
|
+
Check out the [dev app](https://github.com/ankane/blazer-dev) to get started.
|