finery 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +426 -0
- data/CONTRIBUTING.md +49 -0
- data/LICENSE.txt +25 -0
- data/README.md +1144 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.eot +0 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.svg +288 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.ttf +0 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.woff +0 -0
- data/app/assets/fonts/blazer/glyphicons-halflings-regular.woff2 +0 -0
- data/app/assets/images/blazer/favicon.png +0 -0
- data/app/assets/javascripts/blazer/Sortable.js +3709 -0
- data/app/assets/javascripts/blazer/ace/ace.js +19630 -0
- data/app/assets/javascripts/blazer/ace/ext-language_tools.js +1981 -0
- data/app/assets/javascripts/blazer/ace/mode-sql.js +215 -0
- data/app/assets/javascripts/blazer/ace/snippets/sql.js +16 -0
- data/app/assets/javascripts/blazer/ace/snippets/text.js +9 -0
- data/app/assets/javascripts/blazer/ace/theme-twilight.js +18 -0
- data/app/assets/javascripts/blazer/ace.js +6 -0
- data/app/assets/javascripts/blazer/application.js +87 -0
- data/app/assets/javascripts/blazer/bootstrap.js +2580 -0
- data/app/assets/javascripts/blazer/chart.umd.js +13 -0
- data/app/assets/javascripts/blazer/chartjs-adapter-date-fns.bundle.js +6322 -0
- data/app/assets/javascripts/blazer/chartjs-plugin-annotation.min.js +7 -0
- data/app/assets/javascripts/blazer/chartkick.js +2570 -0
- data/app/assets/javascripts/blazer/daterangepicker.js +1578 -0
- data/app/assets/javascripts/blazer/fuzzysearch.js +24 -0
- data/app/assets/javascripts/blazer/highlight.min.js +466 -0
- data/app/assets/javascripts/blazer/jquery.js +10872 -0
- data/app/assets/javascripts/blazer/jquery.stickytableheaders.js +325 -0
- data/app/assets/javascripts/blazer/mapkick.bundle.js +1029 -0
- data/app/assets/javascripts/blazer/moment-timezone-with-data.js +1548 -0
- data/app/assets/javascripts/blazer/moment.js +5685 -0
- data/app/assets/javascripts/blazer/queries.js +130 -0
- data/app/assets/javascripts/blazer/rails-ujs.js +746 -0
- data/app/assets/javascripts/blazer/routes.js +26 -0
- data/app/assets/javascripts/blazer/selectize.js +3891 -0
- data/app/assets/javascripts/blazer/stupidtable-custom-settings.js +13 -0
- data/app/assets/javascripts/blazer/stupidtable.js +281 -0
- data/app/assets/javascripts/blazer/vue.global.prod.js +1 -0
- data/app/assets/stylesheets/blazer/application.css +243 -0
- data/app/assets/stylesheets/blazer/bootstrap-propshaft.css +10 -0
- data/app/assets/stylesheets/blazer/bootstrap-sprockets.css.erb +10 -0
- data/app/assets/stylesheets/blazer/bootstrap.css +6828 -0
- data/app/assets/stylesheets/blazer/daterangepicker.css +410 -0
- data/app/assets/stylesheets/blazer/github.css +125 -0
- data/app/assets/stylesheets/blazer/selectize.css +403 -0
- data/app/controllers/blazer/base_controller.rb +133 -0
- data/app/controllers/blazer/checks_controller.rb +56 -0
- data/app/controllers/blazer/dashboards_controller.rb +99 -0
- data/app/controllers/blazer/queries_controller.rb +468 -0
- data/app/controllers/blazer/uploads_controller.rb +147 -0
- data/app/helpers/blazer/base_helper.rb +83 -0
- data/app/models/blazer/audit.rb +6 -0
- data/app/models/blazer/check.rb +104 -0
- data/app/models/blazer/connection.rb +5 -0
- data/app/models/blazer/dashboard.rb +17 -0
- data/app/models/blazer/dashboard_query.rb +9 -0
- data/app/models/blazer/query.rb +42 -0
- data/app/models/blazer/record.rb +5 -0
- data/app/models/blazer/upload.rb +11 -0
- data/app/models/blazer/uploads_connection.rb +7 -0
- data/app/views/blazer/_nav.html.erb +18 -0
- data/app/views/blazer/_variables.html.erb +127 -0
- data/app/views/blazer/check_mailer/failing_checks.html.erb +7 -0
- data/app/views/blazer/check_mailer/state_change.html.erb +48 -0
- data/app/views/blazer/checks/_form.html.erb +79 -0
- data/app/views/blazer/checks/edit.html.erb +3 -0
- data/app/views/blazer/checks/index.html.erb +72 -0
- data/app/views/blazer/checks/new.html.erb +3 -0
- data/app/views/blazer/dashboards/_form.html.erb +82 -0
- data/app/views/blazer/dashboards/edit.html.erb +3 -0
- data/app/views/blazer/dashboards/new.html.erb +3 -0
- data/app/views/blazer/dashboards/show.html.erb +53 -0
- data/app/views/blazer/queries/_caching.html.erb +16 -0
- data/app/views/blazer/queries/_cohorts.html.erb +48 -0
- data/app/views/blazer/queries/_form.html.erb +255 -0
- data/app/views/blazer/queries/docs.html.erb +147 -0
- data/app/views/blazer/queries/edit.html.erb +2 -0
- data/app/views/blazer/queries/home.html.erb +169 -0
- data/app/views/blazer/queries/new.html.erb +2 -0
- data/app/views/blazer/queries/run.html.erb +188 -0
- data/app/views/blazer/queries/schema.html.erb +55 -0
- data/app/views/blazer/queries/show.html.erb +72 -0
- data/app/views/blazer/uploads/_form.html.erb +27 -0
- data/app/views/blazer/uploads/edit.html.erb +3 -0
- data/app/views/blazer/uploads/index.html.erb +55 -0
- data/app/views/blazer/uploads/new.html.erb +3 -0
- data/app/views/layouts/blazer/application.html.erb +25 -0
- data/config/routes.rb +25 -0
- data/lib/blazer/adapters/athena_adapter.rb +182 -0
- data/lib/blazer/adapters/base_adapter.rb +76 -0
- data/lib/blazer/adapters/bigquery_adapter.rb +79 -0
- data/lib/blazer/adapters/cassandra_adapter.rb +70 -0
- data/lib/blazer/adapters/clickhouse_adapter.rb +84 -0
- data/lib/blazer/adapters/drill_adapter.rb +38 -0
- data/lib/blazer/adapters/druid_adapter.rb +102 -0
- data/lib/blazer/adapters/elasticsearch_adapter.rb +61 -0
- data/lib/blazer/adapters/hive_adapter.rb +55 -0
- data/lib/blazer/adapters/ignite_adapter.rb +64 -0
- data/lib/blazer/adapters/influxdb_adapter.rb +57 -0
- data/lib/blazer/adapters/neo4j_adapter.rb +62 -0
- data/lib/blazer/adapters/opensearch_adapter.rb +52 -0
- data/lib/blazer/adapters/presto_adapter.rb +54 -0
- data/lib/blazer/adapters/salesforce_adapter.rb +50 -0
- data/lib/blazer/adapters/snowflake_adapter.rb +82 -0
- data/lib/blazer/adapters/soda_adapter.rb +105 -0
- data/lib/blazer/adapters/spark_adapter.rb +14 -0
- data/lib/blazer/adapters/sql_adapter.rb +324 -0
- data/lib/blazer/adapters.rb +18 -0
- data/lib/blazer/annotations.rb +47 -0
- data/lib/blazer/anomaly_detectors.rb +22 -0
- data/lib/blazer/check_mailer.rb +27 -0
- data/lib/blazer/data_source.rb +270 -0
- data/lib/blazer/engine.rb +42 -0
- data/lib/blazer/forecasters.rb +7 -0
- data/lib/blazer/result.rb +178 -0
- data/lib/blazer/result_cache.rb +71 -0
- data/lib/blazer/run_statement.rb +44 -0
- data/lib/blazer/run_statement_job.rb +20 -0
- data/lib/blazer/slack_notifier.rb +94 -0
- data/lib/blazer/statement.rb +77 -0
- data/lib/blazer/version.rb +3 -0
- data/lib/blazer.rb +286 -0
- data/lib/finery.rb +3 -0
- data/lib/generators/blazer/install_generator.rb +22 -0
- data/lib/generators/blazer/templates/config.yml.tt +83 -0
- data/lib/generators/blazer/templates/install.rb.tt +47 -0
- data/lib/generators/blazer/templates/uploads.rb.tt +10 -0
- data/lib/generators/blazer/uploads_generator.rb +18 -0
- data/lib/tasks/blazer.rake +20 -0
- data/lib/tasks/finery.rake +20 -0
- data/licenses/LICENSE-ace.txt +24 -0
- data/licenses/LICENSE-bootstrap.txt +21 -0
- data/licenses/LICENSE-chart.js.txt +9 -0
- data/licenses/LICENSE-chartjs-adapter-date-fns.txt +9 -0
- data/licenses/LICENSE-chartkick.js.txt +22 -0
- data/licenses/LICENSE-date-fns.txt +21 -0
- data/licenses/LICENSE-daterangepicker.txt +21 -0
- data/licenses/LICENSE-fuzzysearch.txt +20 -0
- data/licenses/LICENSE-highlight.js.txt +29 -0
- data/licenses/LICENSE-jquery.txt +20 -0
- data/licenses/LICENSE-kurkle-color.txt +9 -0
- data/licenses/LICENSE-mapkick-bundle.txt +1029 -0
- data/licenses/LICENSE-moment-timezone.txt +20 -0
- data/licenses/LICENSE-moment.txt +22 -0
- data/licenses/LICENSE-rails-ujs.txt +20 -0
- data/licenses/LICENSE-selectize.txt +202 -0
- data/licenses/LICENSE-sortable.txt +21 -0
- data/licenses/LICENSE-stickytableheaders.txt +20 -0
- data/licenses/LICENSE-stupidtable.txt +19 -0
- data/licenses/LICENSE-vue.txt +21 -0
- metadata +250 -0
data/README.md
ADDED
|
@@ -0,0 +1,1144 @@
|
|
|
1
|
+
# Finery
|
|
2
|
+
|
|
3
|
+
Explore your data with SQL. Easily create charts and dashboards, and share them with your team.
|
|
4
|
+
|
|
5
|
+
## What exactly is this?
|
|
6
|
+
|
|
7
|
+
Finery is a fork of [Blazer](https://github.com/ankane/blazer) with multiple new features and improvements.
|
|
8
|
+
|
|
9
|
+
I use Blazer daily for personal projects, but I've felt like Blazer PRs take a long time to being merged and I really wanted a more frequently updated drop-in replacement.
|
|
10
|
+
|
|
11
|
+
## Why is it called Finery?
|
|
12
|
+
|
|
13
|
+
I've asked ChatGPT for name suggestions for a gem that is a fork of Blazer, but I wasn't happy with any of the results. Then I've asked it how would you call a fancier blazer and it suggested finery (which Merriam-Webster defines as _ornament, decoration, especially: dressy or showy clothing and jewels_). It's simple to type and easy to pronounce.
|
|
14
|
+
|
|
15
|
+
## Should I send my PR here or to [ankane/blazer](https://github.com/ankane/blazer)?
|
|
16
|
+
|
|
17
|
+
I'm keeping an eye on the open PRs and new code by @ankane. I plan on keeping the full compatibility with Blazer and if that ever changes I'll announce it with at least a couple months in advance.
|
|
18
|
+
|
|
19
|
+
That being said I don't mind direct PRs to this repository, I'll try my best to review them as soon as I can. I reserve the right not to merge everything that got submitted to ankane/blazer, but if it gets accepted there I'll merge it here as well (with some possible changes to match Finery's code).
|
|
20
|
+
|
|
21
|
+
[](https://github.com/finery-bi/finery/actions)
|
|
22
|
+
|
|
23
|
+
## Features
|
|
24
|
+
|
|
25
|
+
- **Multiple data sources** - PostgreSQL, MySQL, Redshift, and [many more](#full-list)
|
|
26
|
+
- **Variables** - run the same queries with different values
|
|
27
|
+
- **Checks & alerts** - get emailed when bad data appears
|
|
28
|
+
- **Audits** - all queries are tracked
|
|
29
|
+
- **Security** - works with your authentication system
|
|
30
|
+
|
|
31
|
+
## Docs
|
|
32
|
+
|
|
33
|
+
- [Installation](#installation)
|
|
34
|
+
- [Queries](#queries)
|
|
35
|
+
- [Charts](#charts)
|
|
36
|
+
- [Dashboards](#dashboards)
|
|
37
|
+
- [Checks](#checks)
|
|
38
|
+
- [Cohorts](#cohorts)
|
|
39
|
+
- [Anomaly Detection](#anomaly-detection)
|
|
40
|
+
- [Forecasting](#forecasting)
|
|
41
|
+
- [Uploads](#uploads)
|
|
42
|
+
- [Data Sources](#data-sources)
|
|
43
|
+
- [Query Permissions](#query-permissions)
|
|
44
|
+
|
|
45
|
+
## Installation
|
|
46
|
+
|
|
47
|
+
Add this line to your application’s Gemfile:
|
|
48
|
+
|
|
49
|
+
```ruby
|
|
50
|
+
gem "finery"
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Run:
|
|
54
|
+
|
|
55
|
+
```sh
|
|
56
|
+
rails generate finery:install
|
|
57
|
+
rails db:migrate
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
And mount the dashboard in your `config/routes.rb`:
|
|
61
|
+
|
|
62
|
+
```ruby
|
|
63
|
+
mount Finery::Engine, at: "finery"
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
For production, specify your database:
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
ENV["BLAZER_DATABASE_URL"] = "postgres://user:password@hostname:5432/database"
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
When possible, Finery tries to protect against queries which modify data by running each query in a transaction and rolling it back, but a safer approach is to use a read-only user. [See how to create one](#permissions).
|
|
73
|
+
|
|
74
|
+
#### Checks (optional)
|
|
75
|
+
|
|
76
|
+
Be sure to set a host in `config/environments/production.rb` for emails to work.
|
|
77
|
+
|
|
78
|
+
```ruby
|
|
79
|
+
config.action_mailer.default_url_options = {host: "finery.dokkuapp.com"}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Schedule checks to run (with cron, [Heroku Scheduler](https://elements.heroku.com/addons/scheduler), etc). The default options are every 5 minutes, 1 hour, or 1 day, which you can customize. For each of these options, set up a task to run.
|
|
83
|
+
|
|
84
|
+
```sh
|
|
85
|
+
rake finery:run_checks SCHEDULE="5 minutes"
|
|
86
|
+
rake finery:run_checks SCHEDULE="1 hour"
|
|
87
|
+
rake finery:run_checks SCHEDULE="1 day"
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
You can also set up failing checks to be sent once a day (or whatever you prefer).
|
|
91
|
+
|
|
92
|
+
```sh
|
|
93
|
+
rake finery:send_failing_checks
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Here’s what it looks like with cron.
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
*/5 * * * * rake finery:run_checks SCHEDULE="5 minutes"
|
|
100
|
+
0 * * * * rake finery:run_checks SCHEDULE="1 hour"
|
|
101
|
+
30 7 * * * rake finery:run_checks SCHEDULE="1 day"
|
|
102
|
+
0 8 * * * rake finery:send_failing_checks
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
For Slack notifications, create an [incoming webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) and set:
|
|
106
|
+
|
|
107
|
+
```sh
|
|
108
|
+
BLAZER_SLACK_WEBHOOK_URL=https://hooks.slack.com/...
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Name the webhook “Finery” and add a cool icon.
|
|
112
|
+
|
|
113
|
+
## Authentication
|
|
114
|
+
|
|
115
|
+
Don’t forget to protect the dashboard in production.
|
|
116
|
+
|
|
117
|
+
### Basic Authentication
|
|
118
|
+
|
|
119
|
+
Set the following variables in your environment or an initializer.
|
|
120
|
+
|
|
121
|
+
```ruby
|
|
122
|
+
ENV["BLAZER_USERNAME"] = "andrew"
|
|
123
|
+
ENV["BLAZER_PASSWORD"] = "secret"
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
### Devise
|
|
127
|
+
|
|
128
|
+
```ruby
|
|
129
|
+
authenticate :user, ->(user) { user.admin? } do
|
|
130
|
+
mount Finery::Engine, at: "finery"
|
|
131
|
+
end
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### Other
|
|
135
|
+
|
|
136
|
+
Specify a `before_action` method to run in `finery.yml`.
|
|
137
|
+
|
|
138
|
+
```yml
|
|
139
|
+
before_action_method: require_admin
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
You can define this method in your `ApplicationController`.
|
|
143
|
+
|
|
144
|
+
```ruby
|
|
145
|
+
def require_admin
|
|
146
|
+
# depending on your auth, something like...
|
|
147
|
+
redirect_to root_path unless current_user && current_user.admin?
|
|
148
|
+
end
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
Be sure to render or redirect for unauthorized users.
|
|
152
|
+
|
|
153
|
+
## Permissions
|
|
154
|
+
|
|
155
|
+
### PostgreSQL
|
|
156
|
+
|
|
157
|
+
Create a user with read-only permissions:
|
|
158
|
+
|
|
159
|
+
```sql
|
|
160
|
+
BEGIN;
|
|
161
|
+
CREATE ROLE finery LOGIN PASSWORD 'secret';
|
|
162
|
+
GRANT CONNECT ON DATABASE dbname TO finery;
|
|
163
|
+
GRANT USAGE ON SCHEMA public TO finery;
|
|
164
|
+
GRANT SELECT ON ALL TABLES IN SCHEMA public TO finery;
|
|
165
|
+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO finery;
|
|
166
|
+
COMMIT;
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
### MySQL
|
|
170
|
+
|
|
171
|
+
Create a user with read-only permissions:
|
|
172
|
+
|
|
173
|
+
```sql
|
|
174
|
+
CREATE USER 'finery'@'127.0.0.1' IDENTIFIED BY 'secret';
|
|
175
|
+
GRANT SELECT, SHOW VIEW ON dbname.* TO 'finery'@'127.0.0.1';
|
|
176
|
+
FLUSH PRIVILEGES;
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
## Sensitive Data
|
|
180
|
+
|
|
181
|
+
If your database contains sensitive or personal data, check out [Hypershield](https://github.com/ankane/hypershield) to shield it.
|
|
182
|
+
|
|
183
|
+
## Encrypted Data
|
|
184
|
+
|
|
185
|
+
If you need to search encrypted data, use [blind indexing](https://github.com/ankane/blind_index).
|
|
186
|
+
|
|
187
|
+
You can have Finery transform specific variables with:
|
|
188
|
+
|
|
189
|
+
```ruby
|
|
190
|
+
Finery.transform_variable = lambda do |name, value|
|
|
191
|
+
value = User.generate_email_bidx(value) if name == "email_bidx"
|
|
192
|
+
value
|
|
193
|
+
end
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
## Queries
|
|
197
|
+
|
|
198
|
+
### Variables
|
|
199
|
+
|
|
200
|
+
Create queries with variables.
|
|
201
|
+
|
|
202
|
+
```sql
|
|
203
|
+
SELECT * FROM users WHERE gender = {gender}
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Use `{start_time}` and `{end_time}` for time ranges. [Example](https://finery.dokkuapp.com/queries/9-time-range-selector?start_time=1997-10-03T05%3A00%3A00%2B00%3A00&end_time=1997-10-04T04%3A59%3A59%2B00%3A00)
|
|
207
|
+
|
|
208
|
+
```sql
|
|
209
|
+
SELECT * FROM ratings WHERE rated_at >= {start_time} AND rated_at <= {end_time}
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
### Smart Variables
|
|
213
|
+
|
|
214
|
+
[Example](https://finery.dokkuapp.com/queries/1-smart-variable)
|
|
215
|
+
|
|
216
|
+
Suppose you have the query:
|
|
217
|
+
|
|
218
|
+
```sql
|
|
219
|
+
SELECT * FROM users WHERE occupation_id = {occupation_id}
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
Instead of remembering each occupation’s id, users can select occupations by name.
|
|
223
|
+
|
|
224
|
+
Add a smart variable with:
|
|
225
|
+
|
|
226
|
+
```yml
|
|
227
|
+
smart_variables:
|
|
228
|
+
occupation_id: "SELECT id, name FROM occupations ORDER BY name ASC"
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
The first column is the value of the variable, and the second column is the label.
|
|
232
|
+
|
|
233
|
+
You can also use an array or hash for static data and enums.
|
|
234
|
+
|
|
235
|
+
```yml
|
|
236
|
+
smart_variables:
|
|
237
|
+
period: ["day", "week", "month"]
|
|
238
|
+
status: {0: "Active", 1: "Archived"}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
### Linked Columns
|
|
242
|
+
|
|
243
|
+
[Example](https://finery.dokkuapp.com/queries/3-linked-column) - title column
|
|
244
|
+
|
|
245
|
+
Link results to other pages in your apps or around the web. Specify a column name and where it should link to. You can use the value of the result with `{value}`.
|
|
246
|
+
|
|
247
|
+
```yml
|
|
248
|
+
linked_columns:
|
|
249
|
+
user_id: "/admin/users/{value}"
|
|
250
|
+
ip_address: "https://www.infosniper.net/index.php?ip_address={value}"
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
### Smart Columns
|
|
254
|
+
|
|
255
|
+
[Example](https://finery.dokkuapp.com/queries/2-smart-column) - occupation_id column
|
|
256
|
+
|
|
257
|
+
Suppose you have the query:
|
|
258
|
+
|
|
259
|
+
```sql
|
|
260
|
+
SELECT name, city_id FROM users
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
See which city the user belongs to without a join.
|
|
264
|
+
|
|
265
|
+
```yml
|
|
266
|
+
smart_columns:
|
|
267
|
+
city_id: "SELECT id, name FROM cities WHERE id IN {value}"
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
You can also use a hash for static data and enums.
|
|
271
|
+
|
|
272
|
+
```yml
|
|
273
|
+
smart_columns:
|
|
274
|
+
status: {0: "Active", 1: "Archived"}
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
### Annotations
|
|
278
|
+
|
|
279
|
+
Shows overlay lines or box ranges for line queries.
|
|
280
|
+
|
|
281
|
+
Suppose your sales data and your deployments data, given a query:
|
|
282
|
+
|
|
283
|
+
```sql
|
|
284
|
+
SELECT date_trunc('hour', created_at), sum(value) FROM sales GROUP BY 1
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
You might want to see the influence of a deployment for those sales.
|
|
288
|
+
|
|
289
|
+
```yml
|
|
290
|
+
annotations:
|
|
291
|
+
deployments: SELECT date, name FROM deployments WHERE date BETWEEN {min_date} AND {max_date}
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
You can also show periods:
|
|
295
|
+
|
|
296
|
+
```yml
|
|
297
|
+
annotations:
|
|
298
|
+
holidays: SELECT min_date, max_date, name FROM holidays WHERE (min_date, max_date) OVERLAPS ({min_date}, {max_date})
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
Conditions for those queries are optional, but they will help to only fetch the relevant annotations for a particular chart.
|
|
302
|
+
|
|
303
|
+
### Caching
|
|
304
|
+
|
|
305
|
+
Finery can automatically cache results to improve speed. It can cache slow queries:
|
|
306
|
+
|
|
307
|
+
```yml
|
|
308
|
+
cache:
|
|
309
|
+
mode: slow
|
|
310
|
+
expires_in: 60 # min
|
|
311
|
+
slow_threshold: 15 # sec
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
Or it can cache all queries:
|
|
315
|
+
|
|
316
|
+
```yml
|
|
317
|
+
cache:
|
|
318
|
+
mode: all
|
|
319
|
+
expires_in: 60 # min
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
Of course, you can force a refresh at any time.
|
|
323
|
+
|
|
324
|
+
## Charts
|
|
325
|
+
|
|
326
|
+
Finery will automatically generate charts based on the types of the columns returned in your query.
|
|
327
|
+
|
|
328
|
+
**Note:** The order of columns matters.
|
|
329
|
+
|
|
330
|
+
### Line Chart
|
|
331
|
+
|
|
332
|
+
There are two ways to generate line charts.
|
|
333
|
+
|
|
334
|
+
2+ columns - timestamp, numeric(s) - [Example](https://finery.dokkuapp.com/queries/4-line-chart-format-1)
|
|
335
|
+
|
|
336
|
+
```sql
|
|
337
|
+
SELECT date_trunc('week', created_at), COUNT(*) FROM users GROUP BY 1
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
3 columns - timestamp, string, numeric - [Example](https://finery.dokkuapp.com/queries/5-line-chart-format-2)
|
|
341
|
+
|
|
342
|
+
|
|
343
|
+
```sql
|
|
344
|
+
SELECT date_trunc('week', created_at), gender, COUNT(*) FROM users GROUP BY 1, 2
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
### Column Chart
|
|
348
|
+
|
|
349
|
+
There are also two ways to generate column charts.
|
|
350
|
+
|
|
351
|
+
2+ columns - string, numeric(s) - [Example](https://finery.dokkuapp.com/queries/6-column-chart-format-1)
|
|
352
|
+
|
|
353
|
+
```sql
|
|
354
|
+
SELECT gender, COUNT(*) FROM users GROUP BY 1
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
3 columns - string, string, numeric - [Example](https://finery.dokkuapp.com/queries/7-column-chart-format-2)
|
|
358
|
+
|
|
359
|
+
```sql
|
|
360
|
+
SELECT gender, zip_code, COUNT(*) FROM users GROUP BY 1, 2
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
### Scatter Chart
|
|
364
|
+
|
|
365
|
+
2 columns - both numeric - [Example](https://finery.dokkuapp.com/queries/16-scatter-chart)
|
|
366
|
+
|
|
367
|
+
```sql
|
|
368
|
+
SELECT x, y FROM table
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
### Pie Chart
|
|
372
|
+
|
|
373
|
+
2 columns - string, numeric - and last column named `pie` - [Example](https://finery.dokkuapp.com/queries/17-pie-chart)
|
|
374
|
+
|
|
375
|
+
```sql
|
|
376
|
+
SELECT gender, COUNT(*) AS pie FROM users GROUP BY 1
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
### Maps
|
|
380
|
+
|
|
381
|
+
Columns named `latitude` and `longitude` or `lat` and `lon` or `lat` and `lng` - [Example](https://finery.dokkuapp.com/queries/15-map)
|
|
382
|
+
|
|
383
|
+
```sql
|
|
384
|
+
SELECT name, latitude, longitude FROM cities
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
or a column named `geojson`
|
|
388
|
+
|
|
389
|
+
```sql
|
|
390
|
+
SELECT name, geojson FROM counties
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
To enable, get an access token from [Mapbox](https://www.mapbox.com/) and set `ENV["MAPBOX_ACCESS_TOKEN"]`.
|
|
394
|
+
|
|
395
|
+
### Targets
|
|
396
|
+
|
|
397
|
+
Use the column name `target` to draw a line for goals. [Example](https://finery.dokkuapp.com/queries/8-target-line)
|
|
398
|
+
|
|
399
|
+
```sql
|
|
400
|
+
SELECT date_trunc('week', created_at), COUNT(*) AS new_users, 100000 AS target FROM users GROUP BY 1
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
## Dashboards
|
|
404
|
+
|
|
405
|
+
Create a dashboard with multiple queries. [Example](https://finery.dokkuapp.com/dashboards/1-dashboard-demo)
|
|
406
|
+
|
|
407
|
+
If the query has a chart, the chart is shown. Otherwise, you’ll see a table.
|
|
408
|
+
|
|
409
|
+
If any queries have variables, they will show up on the dashboard.
|
|
410
|
+
|
|
411
|
+
## Checks
|
|
412
|
+
|
|
413
|
+
Checks give you a centralized place to see the health of your data. [Example](https://finery.dokkuapp.com/checks)
|
|
414
|
+
|
|
415
|
+
Create a query to identify bad rows.
|
|
416
|
+
|
|
417
|
+
```sql
|
|
418
|
+
SELECT * FROM ratings WHERE user_id IS NULL /* all ratings should have a user */
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
Then create check with optional emails if you want to be notified. Emails are sent when a check starts failing, and when it starts passing again.
|
|
422
|
+
|
|
423
|
+
## Cohorts
|
|
424
|
+
|
|
425
|
+
Create a cohort analysis from a simple SQL query. [Example](https://finery.dokkuapp.com/queries/19-cohort-analysis-from-first-order)
|
|
426
|
+
|
|
427
|
+
Create a query with the comment `/* cohort analysis */`. The result should have columns named `user_id` and `conversion_time` and optionally `cohort_time`.
|
|
428
|
+
|
|
429
|
+
You can generate cohorts from the first conversion time:
|
|
430
|
+
|
|
431
|
+
```sql
|
|
432
|
+
/* cohort analysis */
|
|
433
|
+
SELECT user_id, created_at AS conversion_time FROM orders
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
(the first conversion isn’t counted in the first time period with this format)
|
|
437
|
+
|
|
438
|
+
Or from another time, like sign up:
|
|
439
|
+
|
|
440
|
+
```sql
|
|
441
|
+
/* cohort analysis */
|
|
442
|
+
SELECT users.id AS user_id, orders.created_at AS conversion_time, users.created_at AS cohort_time
|
|
443
|
+
FROM users LEFT JOIN orders ON orders.user_id = users.id
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
This feature requires PostgreSQL or MySQL 8.
|
|
447
|
+
|
|
448
|
+
## Anomaly Detection
|
|
449
|
+
|
|
450
|
+
Finery supports three different approaches to anomaly detection.
|
|
451
|
+
|
|
452
|
+
### Prophet
|
|
453
|
+
|
|
454
|
+
Add [prophet-rb](https://github.com/ankane/prophet) to your Gemfile:
|
|
455
|
+
|
|
456
|
+
```ruby
|
|
457
|
+
gem "prophet-rb"
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
And add to `config/finery.yml`:
|
|
461
|
+
|
|
462
|
+
```yml
|
|
463
|
+
anomaly_checks: prophet
|
|
464
|
+
```
|
|
465
|
+
|
|
466
|
+
### Trend
|
|
467
|
+
|
|
468
|
+
[Trend](https://trendapi.org/) uses an external service by default, but you can run it on your own infrastructure as well.
|
|
469
|
+
|
|
470
|
+
Add [trend](https://github.com/ankane/trend) to your Gemfile:
|
|
471
|
+
|
|
472
|
+
```ruby
|
|
473
|
+
gem "trend"
|
|
474
|
+
```
|
|
475
|
+
|
|
476
|
+
And add to `config/finery.yml`:
|
|
477
|
+
|
|
478
|
+
```yml
|
|
479
|
+
anomaly_checks: trend
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
For the [self-hosted API](https://github.com/ankane/trend-api), create an initializer with:
|
|
483
|
+
|
|
484
|
+
```ruby
|
|
485
|
+
Trend.url = "http://localhost:8000"
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
### AnomalyDetection.rb
|
|
489
|
+
|
|
490
|
+
Add [anomaly_detection](https://github.com/ankane/AnomalyDetection.rb) to your Gemfile:
|
|
491
|
+
|
|
492
|
+
```ruby
|
|
493
|
+
gem "anomaly_detection"
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
And add to `config/finery.yml`:
|
|
497
|
+
|
|
498
|
+
```yml
|
|
499
|
+
anomaly_checks: anomaly_detection
|
|
500
|
+
```
|
|
501
|
+
|
|
502
|
+
## Forecasting
|
|
503
|
+
|
|
504
|
+
Finery supports for two different forecasting methods. [Example](https://finery.dokkuapp.com/queries/18-forecast?forecast=t)
|
|
505
|
+
|
|
506
|
+
A forecast link will appear for queries that return 2 columns with types timestamp and numeric.
|
|
507
|
+
|
|
508
|
+
### Prophet
|
|
509
|
+
|
|
510
|
+
Add [prophet-rb](https://github.com/ankane/prophet) to your Gemfile:
|
|
511
|
+
|
|
512
|
+
```ruby
|
|
513
|
+
gem "prophet-rb", ">= 0.2.1"
|
|
514
|
+
```
|
|
515
|
+
|
|
516
|
+
And add to `config/finery.yml`:
|
|
517
|
+
|
|
518
|
+
```yml
|
|
519
|
+
forecasting: prophet
|
|
520
|
+
```
|
|
521
|
+
|
|
522
|
+
### Trend
|
|
523
|
+
|
|
524
|
+
[Trend](https://trendapi.org/) uses an external service by default, but you can run it on your own infrastructure as well.
|
|
525
|
+
|
|
526
|
+
Add [trend](https://github.com/ankane/trend) to your Gemfile:
|
|
527
|
+
|
|
528
|
+
```ruby
|
|
529
|
+
gem "trend"
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
And add to `config/finery.yml`:
|
|
533
|
+
|
|
534
|
+
```yml
|
|
535
|
+
forecasting: trend
|
|
536
|
+
```
|
|
537
|
+
|
|
538
|
+
For the [self-hosted API](https://github.com/ankane/trend-api), create an initializer with:
|
|
539
|
+
|
|
540
|
+
```ruby
|
|
541
|
+
Trend.url = "http://localhost:8000"
|
|
542
|
+
```
|
|
543
|
+
|
|
544
|
+
## Uploads
|
|
545
|
+
|
|
546
|
+
Create database tables from CSV files. [Example](https://finery.dokkuapp.com/uploads)
|
|
547
|
+
|
|
548
|
+
Run:
|
|
549
|
+
|
|
550
|
+
```sh
|
|
551
|
+
rails generate finery:uploads
|
|
552
|
+
rails db:migrate
|
|
553
|
+
```
|
|
554
|
+
|
|
555
|
+
And add to `config/finery.yml`:
|
|
556
|
+
|
|
557
|
+
```yml
|
|
558
|
+
uploads:
|
|
559
|
+
url: postgres://...
|
|
560
|
+
schema: uploads
|
|
561
|
+
data_source: main
|
|
562
|
+
```
|
|
563
|
+
|
|
564
|
+
This feature requires PostgreSQL. Create a new schema just for uploads.
|
|
565
|
+
|
|
566
|
+
```sql
|
|
567
|
+
CREATE SCHEMA uploads;
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
## Data Sources
|
|
571
|
+
|
|
572
|
+
Finery supports multiple data sources :tada:
|
|
573
|
+
|
|
574
|
+
Add additional data sources in `config/finery.yml`:
|
|
575
|
+
|
|
576
|
+
```yml
|
|
577
|
+
data_sources:
|
|
578
|
+
main:
|
|
579
|
+
url: <%= ENV["BLAZER_DATABASE_URL"] %>
|
|
580
|
+
# timeout, smart_variables, linked_columns, smart_columns
|
|
581
|
+
catalog:
|
|
582
|
+
url: <%= ENV["CATALOG_DATABASE_URL"] %>
|
|
583
|
+
# ...
|
|
584
|
+
redshift:
|
|
585
|
+
url: <%= ENV["REDSHIFT_DATABASE_URL"] %>
|
|
586
|
+
# ...
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
### Full List
|
|
590
|
+
|
|
591
|
+
- [Amazon Athena](#amazon-athena)
|
|
592
|
+
- [Amazon Redshift](#amazon-redshift)
|
|
593
|
+
- [Apache Drill](#apache-drill)
|
|
594
|
+
- [Apache Hive](#apache-hive)
|
|
595
|
+
- [Apache Ignite](#apache-ignite)
|
|
596
|
+
- [Apache Spark](#apache-spark)
|
|
597
|
+
- [Cassandra](#cassandra)
|
|
598
|
+
- [ClickHouse](#clickhouse)
|
|
599
|
+
- [Druid](#druid)
|
|
600
|
+
- [Elasticsearch](#elasticsearch)
|
|
601
|
+
- [Google BigQuery](#google-bigquery)
|
|
602
|
+
- [IBM DB2 and Informix](#ibm-db2-and-informix)
|
|
603
|
+
- [InfluxDB](#influxdb)
|
|
604
|
+
- [MySQL](#mysql-1)
|
|
605
|
+
- [Neo4j](#neo4j)
|
|
606
|
+
- [OpenSearch](#opensearch)
|
|
607
|
+
- [Oracle](#oracle)
|
|
608
|
+
- [PostgreSQL](#postgresql-1)
|
|
609
|
+
- [Presto](#presto)
|
|
610
|
+
- [Salesforce](#salesforce)
|
|
611
|
+
- [Socrata Open Data API (SODA)](#socrata-open-data-api-soda)
|
|
612
|
+
- [Snowflake](#snowflake)
|
|
613
|
+
- [SQLite](#sqlite)
|
|
614
|
+
- [SQL Server](#sql-server)
|
|
615
|
+
|
|
616
|
+
You can also [create an adapter](#creating-an-adapter) for any other data store.
|
|
617
|
+
|
|
618
|
+
**Note:** In the examples below, we recommend using environment variables for urls.
|
|
619
|
+
|
|
620
|
+
```yml
|
|
621
|
+
data_sources:
|
|
622
|
+
my_source:
|
|
623
|
+
url: <%= ENV["BLAZER_MY_SOURCE_URL"] %>
|
|
624
|
+
```
|
|
625
|
+
|
|
626
|
+
### Amazon Athena
|
|
627
|
+
|
|
628
|
+
Add [aws-sdk-athena](https://github.com/aws/aws-sdk-ruby) and [aws-sdk-glue](https://github.com/aws/aws-sdk-ruby) to your Gemfile and set:
|
|
629
|
+
|
|
630
|
+
```yml
|
|
631
|
+
data_sources:
|
|
632
|
+
my_source:
|
|
633
|
+
adapter: athena
|
|
634
|
+
database: database
|
|
635
|
+
|
|
636
|
+
# optional settings
|
|
637
|
+
output_location: s3://some-bucket/
|
|
638
|
+
workgroup: primary
|
|
639
|
+
access_key_id: ...
|
|
640
|
+
secret_access_key: ...
|
|
641
|
+
region: ...
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
Here’s an example IAM policy:
|
|
645
|
+
|
|
646
|
+
```json
|
|
647
|
+
{
|
|
648
|
+
"Version": "2012-10-17",
|
|
649
|
+
"Statement": [
|
|
650
|
+
{
|
|
651
|
+
"Effect": "Allow",
|
|
652
|
+
"Action": [
|
|
653
|
+
"athena:GetQueryExecution",
|
|
654
|
+
"athena:GetQueryResults",
|
|
655
|
+
"athena:StartQueryExecution"
|
|
656
|
+
],
|
|
657
|
+
"Resource": [
|
|
658
|
+
"arn:aws:athena:region:account-id:workgroup/primary"
|
|
659
|
+
]
|
|
660
|
+
},
|
|
661
|
+
{
|
|
662
|
+
"Effect": "Allow",
|
|
663
|
+
"Action": [
|
|
664
|
+
"glue:GetTable",
|
|
665
|
+
"glue:GetTables"
|
|
666
|
+
],
|
|
667
|
+
"Resource": [
|
|
668
|
+
"arn:aws:glue:region:account-id:catalog",
|
|
669
|
+
"arn:aws:glue:region:account-id:database/default",
|
|
670
|
+
"arn:aws:glue:region:account-id:table/default/*"
|
|
671
|
+
]
|
|
672
|
+
}
|
|
673
|
+
]
|
|
674
|
+
}
|
|
675
|
+
```
|
|
676
|
+
|
|
677
|
+
You also need to configure [S3 permissions](https://aws.amazon.com/premiumsupport/knowledge-center/access-denied-athena/).
|
|
678
|
+
|
|
679
|
+
### Amazon Redshift
|
|
680
|
+
|
|
681
|
+
Add [activerecord6-redshift-adapter](https://github.com/kwent/activerecord6-redshift-adapter) or [activerecord5-redshift-adapter](https://github.com/ConsultingMD/activerecord5-redshift-adapter) to your Gemfile and set:
|
|
682
|
+
|
|
683
|
+
```yml
|
|
684
|
+
data_sources:
|
|
685
|
+
my_source:
|
|
686
|
+
url: redshift://user:password@hostname:5439/database
|
|
687
|
+
```
|
|
688
|
+
|
|
689
|
+
Use a [read-only user](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html).
|
|
690
|
+
|
|
691
|
+
### Apache Drill
|
|
692
|
+
|
|
693
|
+
Add [drill-sergeant](https://github.com/ankane/drill-sergeant) to your Gemfile and set:
|
|
694
|
+
|
|
695
|
+
```yml
|
|
696
|
+
data_sources:
|
|
697
|
+
my_source:
|
|
698
|
+
adapter: drill
|
|
699
|
+
url: http://hostname:8047
|
|
700
|
+
```
|
|
701
|
+
|
|
702
|
+
Use a [read-only user](https://drill.apache.org/docs/roles-and-privileges/).
|
|
703
|
+
|
|
704
|
+
### Apache Hive
|
|
705
|
+
|
|
706
|
+
Add [hexspace](https://github.com/ankane/hexspace) to your Gemfile and set:
|
|
707
|
+
|
|
708
|
+
```yml
|
|
709
|
+
data_sources:
|
|
710
|
+
my_source:
|
|
711
|
+
adapter: hive
|
|
712
|
+
url: sasl://user:password@hostname:10000/database
|
|
713
|
+
```
|
|
714
|
+
|
|
715
|
+
Use a [read-only user](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization). Requires [HiveServer2](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2).
|
|
716
|
+
|
|
717
|
+
### Apache Ignite
|
|
718
|
+
|
|
719
|
+
Add [ignite-client](https://github.com/ankane/ignite-ruby) to your Gemfile and set:
|
|
720
|
+
|
|
721
|
+
```yml
|
|
722
|
+
data_sources:
|
|
723
|
+
my_source:
|
|
724
|
+
url: ignite://user:password@hostname:10800
|
|
725
|
+
```
|
|
726
|
+
|
|
727
|
+
Use a [read-only user](https://www.gridgain.com/docs/latest/administrators-guide/security/authorization-permissions) (requires a third-party plugin).
|
|
728
|
+
|
|
729
|
+
### Apache Spark
|
|
730
|
+
|
|
731
|
+
Add [hexspace](https://github.com/ankane/hexspace) to your Gemfile and set:
|
|
732
|
+
|
|
733
|
+
```yml
|
|
734
|
+
data_sources:
|
|
735
|
+
my_source:
|
|
736
|
+
adapter: spark
|
|
737
|
+
url: sasl://user:password@hostname:10000/database
|
|
738
|
+
```
|
|
739
|
+
|
|
740
|
+
Use a read-only user. Requires the [Thrift server](https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html).
|
|
741
|
+
|
|
742
|
+
### Cassandra
|
|
743
|
+
|
|
744
|
+
Add [cassandra-driver](https://github.com/datastax/ruby-driver) (and [sorted_set](https://github.com/knu/sorted_set) for Ruby 3+) to your Gemfile and set:
|
|
745
|
+
|
|
746
|
+
```yml
|
|
747
|
+
data_sources:
|
|
748
|
+
my_source:
|
|
749
|
+
url: cassandra://user:password@hostname:9042/keyspace
|
|
750
|
+
```
|
|
751
|
+
|
|
752
|
+
Use a [read-only role](https://docs.datastax.com/en/cql-oss/3.3/cql/cql_using/useSecurePermission.html).
|
|
753
|
+
|
|
754
|
+
### ClickHouse
|
|
755
|
+
|
|
756
|
+
Add [ClickHouse Ruby driver](https://github.com/shlima/click_house) to your Gemfile and set:
|
|
757
|
+
```yml
|
|
758
|
+
data_sources:
|
|
759
|
+
my_source:
|
|
760
|
+
adapter: clickhouse
|
|
761
|
+
url: http://user:password@hostname:8123/database
|
|
762
|
+
|
|
763
|
+
# optional settings
|
|
764
|
+
ssl_verify: true # false by default
|
|
765
|
+
```
|
|
766
|
+
>>>>>>> 41a6b49 (Added support for ClickHouse)
|
|
767
|
+
|
|
768
|
+
### Druid
|
|
769
|
+
|
|
770
|
+
Enable [SQL support](http://druid.io/docs/latest/querying/sql.html#configuration) on the broker and set:
|
|
771
|
+
|
|
772
|
+
```yml
|
|
773
|
+
data_sources:
|
|
774
|
+
my_source:
|
|
775
|
+
adapter: druid
|
|
776
|
+
url: http://hostname:8082
|
|
777
|
+
```
|
|
778
|
+
|
|
779
|
+
Use a [read-only role](https://druid.apache.org/docs/latest/development/extensions-core/druid-basic-security.html).
|
|
780
|
+
|
|
781
|
+
### Elasticsearch
|
|
782
|
+
|
|
783
|
+
Add [elasticsearch](https://github.com/elastic/elasticsearch-ruby) to your Gemfile and set:
|
|
784
|
+
|
|
785
|
+
```yml
|
|
786
|
+
data_sources:
|
|
787
|
+
my_source:
|
|
788
|
+
adapter: elasticsearch
|
|
789
|
+
url: http://user:password@hostname:9200
|
|
790
|
+
```
|
|
791
|
+
|
|
792
|
+
Use a [read-only role](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html).
|
|
793
|
+
|
|
794
|
+
### Google BigQuery
|
|
795
|
+
|
|
796
|
+
Add [google-cloud-bigquery](https://github.com/GoogleCloudPlatform/google-cloud-ruby/tree/master/google-cloud-bigquery) to your Gemfile and set:
|
|
797
|
+
|
|
798
|
+
```yml
|
|
799
|
+
data_sources:
|
|
800
|
+
my_source:
|
|
801
|
+
adapter: bigquery
|
|
802
|
+
project: your-project
|
|
803
|
+
keyfile: path/to/keyfile.json
|
|
804
|
+
```
|
|
805
|
+
|
|
806
|
+
### IBM DB2 and Informix
|
|
807
|
+
|
|
808
|
+
Add [ibm_db](https://github.com/ibmdb/ruby-ibmdb) to your Gemfile and set:
|
|
809
|
+
|
|
810
|
+
```yml
|
|
811
|
+
data_sources:
|
|
812
|
+
my_source:
|
|
813
|
+
url: ibm-db://user:password@hostname:50000/database
|
|
814
|
+
```
|
|
815
|
+
|
|
816
|
+
Use a [read-only user](https://www.ibm.com/support/pages/creating-read-only-database-permissions-user).
|
|
817
|
+
|
|
818
|
+
### InfluxDB
|
|
819
|
+
|
|
820
|
+
Add [influxdb](https://github.com/influxdata/influxdb-ruby) to your Gemfile and set:
|
|
821
|
+
|
|
822
|
+
```yml
|
|
823
|
+
data_sources:
|
|
824
|
+
my_source:
|
|
825
|
+
adapter: influxdb
|
|
826
|
+
url: http://user:password@hostname:8086/database
|
|
827
|
+
```
|
|
828
|
+
|
|
829
|
+
Use a [read-only user](https://docs.influxdata.com/influxdb/v1.8/administration/authentication_and_authorization/). Supports [InfluxQL](https://docs.influxdata.com/influxdb/v1.8/query_language/explore-data/).
|
|
830
|
+
|
|
831
|
+
### MySQL
|
|
832
|
+
|
|
833
|
+
Add [mysql2](https://github.com/brianmario/mysql2) to your Gemfile (if it’s not there) and set:
|
|
834
|
+
|
|
835
|
+
```yml
|
|
836
|
+
data_sources:
|
|
837
|
+
my_source:
|
|
838
|
+
url: mysql2://user:password@hostname:3306/database
|
|
839
|
+
```
|
|
840
|
+
|
|
841
|
+
Use a [read-only user](#mysql).
|
|
842
|
+
|
|
843
|
+
### Neo4j
|
|
844
|
+
|
|
845
|
+
Add [neo4j-core](https://github.com/neo4jrb/neo4j-core) to your Gemfile and set:
|
|
846
|
+
|
|
847
|
+
```yml
|
|
848
|
+
data_sources:
|
|
849
|
+
my_source:
|
|
850
|
+
adapter: neo4j
|
|
851
|
+
url: http://user:password@hostname:7474
|
|
852
|
+
```
|
|
853
|
+
|
|
854
|
+
Use a [read-only user](https://neo4j.com/docs/cypher-manual/current/access-control/manage-privileges/).
|
|
855
|
+
|
|
856
|
+
### OpenSearch
|
|
857
|
+
|
|
858
|
+
Add [opensearch-ruby](https://github.com/opensearch-project/opensearch-ruby) to your Gemfile and set:
|
|
859
|
+
|
|
860
|
+
```yml
|
|
861
|
+
data_sources:
|
|
862
|
+
my_source:
|
|
863
|
+
adapter: opensearch
|
|
864
|
+
url: http://user:password@hostname:9200
|
|
865
|
+
```
|
|
866
|
+
|
|
867
|
+
Use a [read-only user](https://opensearch.org/docs/latest/security-plugin/access-control/permissions/).
|
|
868
|
+
|
|
869
|
+
### Oracle
|
|
870
|
+
|
|
871
|
+
Add [activerecord-oracle_enhanced-adapter](https://github.com/rsim/oracle-enhanced) and [ruby-oci8](https://github.com/kubo/ruby-oci8) to your Gemfile and set:
|
|
872
|
+
|
|
873
|
+
```yml
|
|
874
|
+
data_sources:
|
|
875
|
+
my_source:
|
|
876
|
+
url: oracle-enhanced://user:password@hostname:1521/database
|
|
877
|
+
```
|
|
878
|
+
|
|
879
|
+
Use a [read-only user](https://docs.oracle.com/cd/B19306_01/network.102/b14266/authoriz.htm).
|
|
880
|
+
|
|
881
|
+
### PostgreSQL
|
|
882
|
+
|
|
883
|
+
Add [pg](https://github.com/ged/ruby-pg) to your Gemfile (if it’s not there) and set:
|
|
884
|
+
|
|
885
|
+
```yml
|
|
886
|
+
data_sources:
|
|
887
|
+
my_source:
|
|
888
|
+
url: postgres://user:password@hostname:5432/database
|
|
889
|
+
```
|
|
890
|
+
|
|
891
|
+
Use a [read-only user](#postgresql).
|
|
892
|
+
|
|
893
|
+
### Presto
|
|
894
|
+
|
|
895
|
+
Add [presto-client](https://github.com/treasure-data/presto-client-ruby) to your Gemfile and set:
|
|
896
|
+
|
|
897
|
+
```yml
|
|
898
|
+
data_sources:
|
|
899
|
+
my_source:
|
|
900
|
+
url: presto://user@hostname:8080/catalog
|
|
901
|
+
```
|
|
902
|
+
|
|
903
|
+
Use a [read-only user](https://prestodb.io/docs/current/security/built-in-system-access-control.html).
|
|
904
|
+
|
|
905
|
+
### Salesforce
|
|
906
|
+
|
|
907
|
+
Add [restforce](https://github.com/restforce/restforce) to your Gemfile and set:
|
|
908
|
+
|
|
909
|
+
```yml
|
|
910
|
+
data_sources:
|
|
911
|
+
my_source:
|
|
912
|
+
adapter: salesforce
|
|
913
|
+
```
|
|
914
|
+
|
|
915
|
+
And set the appropriate environment variables:
|
|
916
|
+
|
|
917
|
+
```sh
|
|
918
|
+
SALESFORCE_USERNAME="username"
|
|
919
|
+
SALESFORCE_PASSWORD="password"
|
|
920
|
+
SALESFORCE_SECURITY_TOKEN="security token"
|
|
921
|
+
SALESFORCE_CLIENT_ID="client id"
|
|
922
|
+
SALESFORCE_CLIENT_SECRET="client secret"
|
|
923
|
+
SALESFORCE_API_VERSION="41.0"
|
|
924
|
+
```
|
|
925
|
+
|
|
926
|
+
Use a read-only user. Supports [SOQL](https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql.htm).
|
|
927
|
+
|
|
928
|
+
### Socrata Open Data API (SODA)
|
|
929
|
+
|
|
930
|
+
Set:
|
|
931
|
+
|
|
932
|
+
```yml
|
|
933
|
+
data_sources:
|
|
934
|
+
my_source:
|
|
935
|
+
adapter: soda
|
|
936
|
+
url: https://soda.demo.socrata.com/resource/4tka-6guv.json
|
|
937
|
+
app_token: ...
|
|
938
|
+
```
|
|
939
|
+
|
|
940
|
+
Supports [SoQL](https://dev.socrata.com/docs/functions/).
|
|
941
|
+
|
|
942
|
+
### Snowflake
|
|
943
|
+
|
|
944
|
+
First, install ODBC. For Homebrew, use:
|
|
945
|
+
|
|
946
|
+
```sh
|
|
947
|
+
brew install unixodbc
|
|
948
|
+
```
|
|
949
|
+
|
|
950
|
+
For Ubuntu, use:
|
|
951
|
+
|
|
952
|
+
```sh
|
|
953
|
+
sudo apt-get install unixodbc-dev
|
|
954
|
+
```
|
|
955
|
+
|
|
956
|
+
For Heroku, use the [Apt buildpack](https://github.com/heroku/heroku-buildpack-apt) and create an `Aptfile` with:
|
|
957
|
+
|
|
958
|
+
```text
|
|
959
|
+
unixodbc-dev
|
|
960
|
+
https://sfc-repo.snowflakecomputing.com/odbc/linux/2.21.5/snowflake-odbc-2.21.5.x86_64.deb
|
|
961
|
+
```
|
|
962
|
+
|
|
963
|
+
> This installs the driver at `/app/.apt/usr/lib/snowflake/odbc/lib/libSnowflake.so`
|
|
964
|
+
|
|
965
|
+
Then, download the [Snowflake ODBC driver](https://docs.snowflake.net/manuals/user-guide/odbc-download.html). Add [odbc_adapter](https://github.com/localytics/odbc_adapter) to your Gemfile and set:
|
|
966
|
+
|
|
967
|
+
```yml
|
|
968
|
+
data_sources:
|
|
969
|
+
my_source:
|
|
970
|
+
adapter: snowflake
|
|
971
|
+
conn_str: Driver=/path/to/libSnowflake.so;uid=user;pwd=password;server=host.snowflakecomputing.com
|
|
972
|
+
```
|
|
973
|
+
|
|
974
|
+
Use a [read-only role](https://docs.snowflake.com/en/user-guide/security-access-control-configure.html).
|
|
975
|
+
|
|
976
|
+
### SQLite
|
|
977
|
+
|
|
978
|
+
Add [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) to your Gemfile and set:
|
|
979
|
+
|
|
980
|
+
```yml
|
|
981
|
+
data_sources:
|
|
982
|
+
my_source:
|
|
983
|
+
url: sqlite3:path/to/database.sqlite3
|
|
984
|
+
```
|
|
985
|
+
|
|
986
|
+
### SQL Server
|
|
987
|
+
|
|
988
|
+
Add [tiny_tds](https://github.com/rails-sqlserver/tiny_tds) and [activerecord-sqlserver-adapter](https://github.com/rails-sqlserver/activerecord-sqlserver-adapter) to your Gemfile and set:
|
|
989
|
+
|
|
990
|
+
```yml
|
|
991
|
+
data_sources:
|
|
992
|
+
my_source:
|
|
993
|
+
url: sqlserver://user:password@hostname:1433/database
|
|
994
|
+
```
|
|
995
|
+
|
|
996
|
+
Use a [read-only user](https://docs.microsoft.com/en-us/sql/relational-databases/security/authentication-access/getting-started-with-database-engine-permissions?view=sql-server-ver15).
|
|
997
|
+
|
|
998
|
+
## Creating an Adapter
|
|
999
|
+
|
|
1000
|
+
Create an adapter for any data store with:
|
|
1001
|
+
|
|
1002
|
+
```ruby
|
|
1003
|
+
class FooAdapter < Finery::Adapters::BaseAdapter
|
|
1004
|
+
# code goes here
|
|
1005
|
+
end
|
|
1006
|
+
|
|
1007
|
+
Finery.register_adapter "foo", FooAdapter
|
|
1008
|
+
```
|
|
1009
|
+
|
|
1010
|
+
See the [Presto adapter](https://github.com/finery-bi/finery/blob/master/lib/finery/adapters/presto_adapter.rb) for a good example. Then use:
|
|
1011
|
+
|
|
1012
|
+
```yml
|
|
1013
|
+
data_sources:
|
|
1014
|
+
my_source:
|
|
1015
|
+
adapter: foo
|
|
1016
|
+
url: http://user:password@hostname:9200/
|
|
1017
|
+
```
|
|
1018
|
+
|
|
1019
|
+
## Query Permissions
|
|
1020
|
+
|
|
1021
|
+
Finery supports a basic permissions model.
|
|
1022
|
+
|
|
1023
|
+
1. Queries without a name are unlisted
|
|
1024
|
+
2. Queries whose name starts with `#` are only listed to the creator
|
|
1025
|
+
3. Queries whose name starts with `*` can only be edited by the creator
|
|
1026
|
+
|
|
1027
|
+
## Learn SQL
|
|
1028
|
+
|
|
1029
|
+
Have team members who want to learn SQL? Here are a few great, free resources.
|
|
1030
|
+
|
|
1031
|
+
- [The Data School](https://dataschool.com/learn-sql/)
|
|
1032
|
+
- [SQLBolt](https://sqlbolt.com/)
|
|
1033
|
+
|
|
1034
|
+
## Useful Tools
|
|
1035
|
+
|
|
1036
|
+
For an easy way to group by day, week, month, and more with correct time zones, check out [Groupdate.sql](https://github.com/ankane/groupdate.sql).
|
|
1037
|
+
|
|
1038
|
+
## Standalone Version
|
|
1039
|
+
|
|
1040
|
+
Looking for a standalone version? Check out [Ghost Finery](https://github.com/buren/ghost_finery).
|
|
1041
|
+
|
|
1042
|
+
## Performance
|
|
1043
|
+
|
|
1044
|
+
By default, queries take up a request while they are running. To run queries asynchronously, add to your config:
|
|
1045
|
+
|
|
1046
|
+
```yml
|
|
1047
|
+
async: true
|
|
1048
|
+
```
|
|
1049
|
+
|
|
1050
|
+
**Note:** Requires caching to be enabled. If you have multiple web processes, your app must use a centralized cache store like Memcached or Redis.
|
|
1051
|
+
|
|
1052
|
+
```ruby
|
|
1053
|
+
config.cache_store = :mem_cache_store
|
|
1054
|
+
```
|
|
1055
|
+
|
|
1056
|
+
## Archiving
|
|
1057
|
+
|
|
1058
|
+
Archive queries that haven’t been viewed in over 90 days.
|
|
1059
|
+
|
|
1060
|
+
```sh
|
|
1061
|
+
rake finery:archive_queries
|
|
1062
|
+
```
|
|
1063
|
+
|
|
1064
|
+
## Content Security Policy
|
|
1065
|
+
|
|
1066
|
+
If views are stuck with a `Loading...` message, there might be a problem with strict CSP settings in your app. This can be checked with Firefox or Chrome dev tools. You can allow Finery to override these settings for its controllers with:
|
|
1067
|
+
|
|
1068
|
+
```yml
|
|
1069
|
+
override_csp: true
|
|
1070
|
+
```
|
|
1071
|
+
|
|
1072
|
+
## Upgrading
|
|
1073
|
+
|
|
1074
|
+
Note: Finery started as a fork of Blazer 3.0, so everything before that is strictly Blazer-only.
|
|
1075
|
+
|
|
1076
|
+
### 3.0
|
|
1077
|
+
|
|
1078
|
+
Maps now use Mapbox GL JS v1 instead of Mapbox.js, which affects Mapbox billing.
|
|
1079
|
+
|
|
1080
|
+
### 2.6
|
|
1081
|
+
|
|
1082
|
+
Custom adapters now need to specify how to quote variables in queries (there is no longer a default)
|
|
1083
|
+
|
|
1084
|
+
```ruby
|
|
1085
|
+
class FooAdapter < Finery::Adapters::BaseAdapter
|
|
1086
|
+
def quoting
|
|
1087
|
+
:backslash_escape # single quote strings and convert ' to \' and \ to \\
|
|
1088
|
+
# or
|
|
1089
|
+
:single_quote_escape # single quote strings and convert ' to ''
|
|
1090
|
+
# or
|
|
1091
|
+
->(value) { ... } # custom method
|
|
1092
|
+
end
|
|
1093
|
+
end
|
|
1094
|
+
```
|
|
1095
|
+
|
|
1096
|
+
### 2.3
|
|
1097
|
+
|
|
1098
|
+
To archive queries, create a migration
|
|
1099
|
+
|
|
1100
|
+
```sh
|
|
1101
|
+
rails g migration add_status_to_finery_queries
|
|
1102
|
+
```
|
|
1103
|
+
|
|
1104
|
+
with:
|
|
1105
|
+
|
|
1106
|
+
```ruby
|
|
1107
|
+
add_column :finery_queries, :status, :string
|
|
1108
|
+
Finery::Query.update_all(status: "active")
|
|
1109
|
+
```
|
|
1110
|
+
|
|
1111
|
+
### 2.0
|
|
1112
|
+
|
|
1113
|
+
To use Slack notifications, create a migration
|
|
1114
|
+
|
|
1115
|
+
```sh
|
|
1116
|
+
rails g migration add_slack_channels_to_finery_checks
|
|
1117
|
+
```
|
|
1118
|
+
|
|
1119
|
+
with:
|
|
1120
|
+
|
|
1121
|
+
```ruby
|
|
1122
|
+
add_column :finery_checks, :slack_channels, :text
|
|
1123
|
+
```
|
|
1124
|
+
|
|
1125
|
+
## History
|
|
1126
|
+
|
|
1127
|
+
View the [changelog](https://github.com/finery-bi/finery/blob/master/CHANGELOG.md)
|
|
1128
|
+
|
|
1129
|
+
## Thanks
|
|
1130
|
+
|
|
1131
|
+
Finery uses a number of awesome open source projects, including [Rails](https://github.com/rails/rails/), [Vue.js](https://github.com/vuejs/vue), [jQuery](https://github.com/jquery/jquery), [Bootstrap](https://github.com/twbs/bootstrap), [Selectize](https://github.com/brianreavis/selectize.js), [StickyTableHeaders](https://github.com/jmosbech/StickyTableHeaders), [Stupid jQuery Table Sort](https://github.com/joequery/Stupid-Table-Plugin), and [Date Range Picker](https://github.com/dangrossman/bootstrap-daterangepicker).
|
|
1132
|
+
|
|
1133
|
+
Demo data from [MovieLens](https://grouplens.org/datasets/movielens/).
|
|
1134
|
+
|
|
1135
|
+
## Want to Make Finery Better?
|
|
1136
|
+
|
|
1137
|
+
That’s awesome! Here are a few ways you can help:
|
|
1138
|
+
|
|
1139
|
+
- [Report bugs](https://github.com/finery-bi/finery/issues)
|
|
1140
|
+
- Fix bugs and [submit pull requests](https://github.com/finery-bi/finery/pulls)
|
|
1141
|
+
- Write, clarify, or fix documentation
|
|
1142
|
+
- Suggest or add new features
|
|
1143
|
+
|
|
1144
|
+
Check out the [dev app](https://github.com/finery-bi/finery-dev) to get started.
|