beetle_etl 0.0.19 → 1.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dacfd8801cf603b9cec442ee9ea025bc14026234
-  data.tar.gz: 1a183e975200bb9bdc6e631130903b6ead9739be
+  metadata.gz: 0c5a5a3f4d0b4170ccb1581a50a2fccaf32be2cb
+  data.tar.gz: c8beb0f668010410cadefe1ae7ab7c7708032acc
 SHA512:
-  metadata.gz: bacbe3f6292e52b1ad23d38f5e128b1cf39ec63ded808310a8353956f2b11d349b4ea546012beb22e03edbd3de9bdd787ca0bbde329e1755ebacc23c9561a8ca
-  data.tar.gz: cbc51b9be8c35b70c21829af0c38f6db0898f4b5c38a2fc6138968bc63591e91f51d7558b2fff62196ba43789b3255f2347c9982d78de33a4d025e3da167084d
+  metadata.gz: 3b88ee3cd93eb9344b95bc0b31e01da0a1419d90115fe7a435541b8a336cc3eef8dbad68df819f9993dab3f326be7a91f7b89f8f1e3133b81584834c8f973d65
+  data.tar.gz: e696f99dee13095e6ac7ee32a4fc7e0364744bafbc28caca41b8ab5ce87af3c66e832493b116070545401e9e43bbc34d7582f9d004991beac6b994d3a2d2a9f4

data/README.md

@@ -2,7 +2,11 @@
 [![Build Status](https://travis-ci.org/maiwald/beetle_etl.svg?branch=master)](https://travis-ci.org/maiwald/beetle_etl)
 [![Code Climate](https://codeclimate.com/github/maiwald/beetle_etl.png)](https://codeclimate.com/github/maiwald/beetle_etl)
 
-BeetleETL helps you with synchronising relational databases and recurring imports of data. It is actually quite nice.
+BeetleETL helps you with synchronising relational databases and recurring imports of reference data. It is actually quite nice.
+
+Consider you have a set of database tables representing third party data (i.e. the ```source```) and you want to synchronize a set of tables in your application (i.e. the ```target```) with that third party data. Further consider that you want to apply transformations to that ```source``` data before you import it.
+
+You define your transformations and BeetleETL will to the rest. Even when your ```source``` data changes, when you run BeetleETL again, it can keep track of what changes need to be applied to what records in your application’s tables.
 
 It currently only works with PostgreSQL databases.
 
@@ -20,21 +24,25 @@ Or install it yourself as:
 
     $ gem install beetle_etl
 
+## Setup
+
+Make sure the tables you want to import contain columns named ```external_id``` and ```external_source``` of type ```CHARACTER VARYING(255)```, as well as timestamp columns named ```created_at```, ```updated_at``` and, ```deleted_at```.
+
 ## Usage
 
 ### Configuration
 
     BeetleETL.configure do |config|
-      config.transformation_file = # path to your imports
+      config.transformation_file = # path to your transformation file
       config.database_config = # sequel database config
     # or config.database = # sequel database instance
-      config.external_source = ‘source_name’
+      config.external_source = ‘name_of_your_source’
       config.logger = Logger.new(STDOUT)
     end
 
 ### Defining Imports
 
-Fill a file with all the tables you wish to import and write queries to select the data you want.
+Fill a ```transformation``` file with import directives like this:
 
     import :departments do
       columns :name
@@ -58,6 +66,12 @@ Fill a file with all the tables you wish to import and write queries to select t
           ON data.org_id = o.id
       SQL
     end
+    
+    
+```import``` takes the name of the table you want to fill and the configuration as arguments.
+With ```columns``` you define what columns BeetleETL is supposed to fill in your application’s table.
+The ```query``` transforms the data. Make sure that you insert into ```#{stage_table}``` as the name of the actual table, that this inserts into will be filled in by BeetleETL during runtime.
+Define any foreign references your table has to other tables using the ```refrecences(on:)``` directive. For every foreign key your table has, BeeteETL requires you to fill in a column named ```external_foreign_key``` (prepend "```external_```" to your actual foreign key column).
 
 
 ### Running BeetleETL

data/lib/beetle_etl/step_runner/async_step_runner.rb

@@ -58,7 +58,7 @@ module BeetleETL
           BeetleETL.logger.fatal(e.message)
           raise e
         end
-      end
+      end.abort_on_exception = true
     end
 
     def mark_step_running(step_name)

data/lib/beetle_etl/version.rb

@@ -1,3 +1,3 @@
 module BeetleETL
-  VERSION = "0.0.19"
+  VERSION = "1.0.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: beetle_etl
 version: !ruby/object:Gem::Version
-  version: 0.0.19
+  version: 1.0.1
 platform: ruby
 authors:
 - Luciano Maiwald
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-04-26 00:00:00.000000000 Z
+date: 2015-09-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sequel