beetle_etl 0.0.19 → 1.0.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +18 -4
- data/lib/beetle_etl/step_runner/async_step_runner.rb +1 -1
- data/lib/beetle_etl/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 0c5a5a3f4d0b4170ccb1581a50a2fccaf32be2cb
|
4
|
+
data.tar.gz: c8beb0f668010410cadefe1ae7ab7c7708032acc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3b88ee3cd93eb9344b95bc0b31e01da0a1419d90115fe7a435541b8a336cc3eef8dbad68df819f9993dab3f326be7a91f7b89f8f1e3133b81584834c8f973d65
|
7
|
+
data.tar.gz: e696f99dee13095e6ac7ee32a4fc7e0364744bafbc28caca41b8ab5ce87af3c66e832493b116070545401e9e43bbc34d7582f9d004991beac6b994d3a2d2a9f4
|
data/README.md
CHANGED
@@ -2,7 +2,11 @@
|
|
2
2
|
[![Build Status](https://travis-ci.org/maiwald/beetle_etl.svg?branch=master)](https://travis-ci.org/maiwald/beetle_etl)
|
3
3
|
[![Code Climate](https://codeclimate.com/github/maiwald/beetle_etl.png)](https://codeclimate.com/github/maiwald/beetle_etl)
|
4
4
|
|
5
|
-
BeetleETL helps you with synchronising relational databases and recurring imports of data. It is actually quite nice.
|
5
|
+
BeetleETL helps you with synchronising relational databases and recurring imports of reference data. It is actually quite nice.
|
6
|
+
|
7
|
+
Consider you have a set of database tables representing third party data (i.e. the ```source```) and you want to synchronize a set of tables in your application (i.e. the ```target```) with that third party data. Further consider that you want to apply transformations to that ```source``` data before you import it.
|
8
|
+
|
9
|
+
You define your transformations and BeetleETL will to the rest. Even when your ```source``` data changes, when you run BeetleETL again, it can keep track of what changes need to be applied to what records in your application’s tables.
|
6
10
|
|
7
11
|
It currently only works with PostgreSQL databases.
|
8
12
|
|
@@ -20,21 +24,25 @@ Or install it yourself as:
|
|
20
24
|
|
21
25
|
$ gem install beetle_etl
|
22
26
|
|
27
|
+
## Setup
|
28
|
+
|
29
|
+
Make sure the tables you want to import contain columns named ```external_id``` and ```external_source``` of type ```CHARACTER VARYING(255)```, as well as timestamp columns named ```created_at```, ```updated_at``` and, ```deleted_at```.
|
30
|
+
|
23
31
|
## Usage
|
24
32
|
|
25
33
|
### Configuration
|
26
34
|
|
27
35
|
BeetleETL.configure do |config|
|
28
|
-
config.transformation_file = # path to your
|
36
|
+
config.transformation_file = # path to your transformation file
|
29
37
|
config.database_config = # sequel database config
|
30
38
|
# or config.database = # sequel database instance
|
31
|
-
config.external_source = ‘
|
39
|
+
config.external_source = ‘name_of_your_source’
|
32
40
|
config.logger = Logger.new(STDOUT)
|
33
41
|
end
|
34
42
|
|
35
43
|
### Defining Imports
|
36
44
|
|
37
|
-
Fill a file with
|
45
|
+
Fill a ```transformation``` file with import directives like this:
|
38
46
|
|
39
47
|
import :departments do
|
40
48
|
columns :name
|
@@ -58,6 +66,12 @@ Fill a file with all the tables you wish to import and write queries to select t
|
|
58
66
|
ON data.org_id = o.id
|
59
67
|
SQL
|
60
68
|
end
|
69
|
+
|
70
|
+
|
71
|
+
```import``` takes the name of the table you want to fill and the configuration as arguments.
|
72
|
+
With ```columns``` you define what columns BeetleETL is supposed to fill in your application’s table.
|
73
|
+
The ```query``` transforms the data. Make sure that you insert into ```#{stage_table}``` as the name of the actual table, that this inserts into will be filled in by BeetleETL during runtime.
|
74
|
+
Define any foreign references your table has to other tables using the ```refrecences(on:)``` directive. For every foreign key your table has, BeeteETL requires you to fill in a column named ```external_foreign_key``` (prepend "```external_```" to your actual foreign key column).
|
61
75
|
|
62
76
|
|
63
77
|
### Running BeetleETL
|
data/lib/beetle_etl/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: beetle_etl
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 1.0.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Luciano Maiwald
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-
|
11
|
+
date: 2015-09-16 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: sequel
|