samidare 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f97f6368bca22b657df37c1279d7b5d07c514c76
-  data.tar.gz: 62f4b01e91cfda7c95b787e71a047e5d8fb2155f
+  metadata.gz: 9bc2846461bb56c026fab7855d4fa3bbb5d9a6fa
+  data.tar.gz: fec1ae897422431b75114117f89e4c442d3fa1ac
 SHA512:
-  metadata.gz: 64dde2e0f6fd33eae7b2d044acabe546ee097745617410a751b6dfa48e3fd8bd8389d80e820037ec876b299e74637ef6ea1db22dbe56e793b3a55c533e7ef166
-  data.tar.gz: 1c687b2b61a237444ce5d7e9f1dbf44d73985482a0ef8d31178806a1321a11224b97277d756c26ffffb891f87689e57335e0772ae913340805a5418c77b14ada
+  metadata.gz: 77c2030170b64c5006f0d6a773bf58c365dd1b4f1261028ffe8171058f5101bf72c71934278def78147952aae7d4aff6eb7f88131eaca56c0de12aaafb1e9438
+  data.tar.gz: 0cd7332f0bf04d0f2d3373411b46c7c3495f9c31dc6464fb934abb154e31fa552b76612feadc62cdc8c431618a389d314400db586cff1fb94260698d88f57e6e

data/.gitignore

@@ -1,15 +1,15 @@
-/.bundle/
-/.yardoc
-/Gemfile.lock
-/_yardoc/
-/coverage/
-/doc/
-/pkg/
-/spec/reports/
-/tmp/
-*.bundle
-*.so
-*.o
-*.a
-.rspec
-mkmf.log
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+.rspec
+mkmf.log

data/README.md

@@ -1,117 +1,179 @@
-# Samidare
-
-Generate Embulk config and BigQuery schema from MySQL schema and run Embulk.
-
-## Installation
-
-Add this line to your application's Gemfile:
-
-```ruby
-gem 'samidare'
-```
-
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install samidare
-
-## Usage
-Require `database.yml` and `table.yml`.
-Below is a sample config file.
-
-#### database.yml
-```yml
-db01:
-  host: localhost
-  username: root
-  password: pswd
-  database: production
-  bq_dataset: mysql_db01
-
-db02:
-  host: localhost
-  username: root
-  password: pswd
-  database: production
-  bq_dataset: mysql_db02
-
-```
-
-**Caution: Embulk doesn't allow no password for MySQL**
-
-#### table.yml
-```yml
-db01:
-  tables:
-    - name: users
-    - name: events
-    - name: hobbies
-
-db02:
-  tables:
-    - name: administrators
-    - name: configs
-```
-
-Samidare requires BigQuery parameters like below.
-```ruby
-[sample.rb]
-config = {
- 'project_id' => 'BIGQUERY_PROJECT_ID',
- 'service_email' => 'SERVICE_ACCOUNT_EMAIL',
- 'key' => '/etc/embulk/bigquery.p12',
- 'schema_dir' => '/var/tmp/embulk/schema',
- 'config_dir' => '/var/tmp/embulk/config',
- 'auth_method' => 'private_key'
-}
-
-client = Samidare::EmbulkClient.new
-client.generate_config(config)
-client.run(config)
-```
-
-```bash
-bundle exec ruby sample.rb
-```
-
-## Features
-#### daily snapshot
-BigQuery supports table wildcard expression of a specific set of daily tables, for example, `sales20150701` .  
-If you need daily snapshot of a table for BigQuery, use `daily_snapshot` option to `database.yml` or `table.yml` like below.  
-`daily_snapshot` option effects all tables in case of  `database.yml` .  
-On the other hand, only target table in `table.yml` .  
-**Daily part is determined by execute date.**
-
-```yml
-[database.yml]
-production:
-  host: localhost
-  username: root
-  password: pswd
-  database: production
-  bq_dataset: mysql
-  daily_snapshot: true
-```
-
-```yml
-[table.yml]
-production:
-  tables:
-    - name: users
-    - name: events
-      daily_snapshot: true
-    - name: hobbies
-
-Only `events` is renamed to `eventsYYYYMMDD` for BigQuery.
-```
-
-## Contributing
-
-1. Fork it ( https://github.com/[my-github-username]/samidare/fork )
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create a new Pull Request
+# Samidare
+
+Generate Embulk config and BigQuery schema from MySQL schema and run Embulk.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'samidare'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install samidare
+
+## Embulk setup
+`Samidare` is utility for `Embulk` .
+You need to install `Embulk` and install some gems like below.
+
+```bash
+embulk gem install embulk-input-mysql --version 0.8.2
+embulk gem install embulk-output-bigquery --version 0.4.3
+embulk gem install embulk-parser-jsonl --version 0.2.0
+embulk gem install embulk-formatter-jsonl --version 0.1.4
+```
+
+## Usage
+Require `database.yml` and `table.yml`.
+Below is a sample config file.
+
+### database.yml
+```yml
+db01:
+  host: localhost
+  username: root
+  password: pswd
+  database: production
+  bq_dataset: mysql_db01
+
+db02:
+  host: localhost
+  username: root
+  password: pswd
+  database: production
+  bq_dataset: mysql_db02
+
+```
+
+**Caution: Embulk doesn't allow no password for MySQL**
+
+### table.yml
+```yml
+db01:
+  tables:
+    - name: users
+    - name: events
+    - name: hobbies
+
+db02:
+  tables:
+    - name: administrators
+    - name: configs
+```
+
+Samidare requires BigQuery parameters like below.
+
+```ruby
+[sample.rb]
+require 'samidare'
+
+config = {
+ 'project_id' => 'BIGQUERY_PROJECT_ID',
+ 'service_email' => 'SERVICE_ACCOUNT_EMAIL',
+ 'key' => '/etc/embulk/bigquery.p12',
+ 'schema_dir' => '/var/tmp/embulk/schema',
+ 'config_dir' => '/var/tmp/embulk/config',
+ 'auth_method' => 'private_key'
+}
+
+client = Samidare::EmbulkClient.new
+client.generate_config(config)
+client.run(config)
+```
+
+```bash
+ruby sample.rb
+```
+
+## Features
+### process status
+`Samidare` returns process status as boolean.  
+If all tables are succeed, then returns `true`, else `false` .  
+It is useful to control system flow.
+
+```ruby
+process_status = Samidare::EmbulkClient.new.run(config)
+exit 1 unless process_status
+```
+
+### narrow tables
+You can narrow actual target tables from `table.yml` for test or to retry.  
+If no target tables is given, `Samidare` will execute all tables.
+
+```ruby
+# in case, all tables are ['users', 'purchases', 'items']
+target_tables = ['users', 'purchases']
+Samidare::EmbulkClient.new.run(config, target_tables)
+```
+
+### retry
+You can set retry count.  
+If any table failed, only failed table will be retried until retry count.  
+If no retry count is given, `Samidare` dosen't retry.
+
+```ruby
+# 2 times retry will execute
+Samidare::EmbulkClient.new.run(config, [], 2)
+```
+
+### SQL condition
+If you set `condition` to a table in `table.yml` , SQL is generated like below.  
+It is useful for large size table.
+
+```yml
+[table.yml]
+production:
+  tables:
+    - name: users
+    - name: events
+      conditon: created_at < CURRENT_DATE()
+```
+
+```sql
+SELECT * FROM users
+SELECT * FROM events WHERE created_at < CURRENT_DATE()
+```
+
+### daily snapshot
+BigQuery supports table wildcard expression of a specific set of daily tables, for example, `sales20150701` .  
+If you need daily snapshot of a table for BigQuery, use `daily_snapshot` option to `database.yml` or `table.yml` like below.  
+`daily_snapshot` option effects all tables in case of  `database.yml` .  
+On the other hand, only target table in `table.yml` .  
+**Daily part is determined by execute date.**
+
+```yml
+[database.yml]
+production:
+  host: localhost
+  username: root
+  password: pswd
+  database: production
+  bq_dataset: mysql
+  daily_snapshot: true
+```
+
+```yml
+[table.yml]
+production:
+  tables:
+    - name: users
+    - name: events
+      daily_snapshot: true
+    - name: hobbies
+
+Only `events` is renamed to `eventsYYYYMMDD` for BigQuery.
+```
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/samidare/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -1,7 +1,7 @@
-require "bundler/gem_tasks"
-require "rspec/core/rake_task"
-
-RSpec::Core::RakeTask.new(:spec)
-
-task :default => :spec
-
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+

data/lib/samidare.rb

@@ -1,42 +1,42 @@
-require 'samidare/version'
-require 'samidare/embulk_utility'
-require 'samidare/embulk'
-require 'samidare/mysql'
-
-module Samidare
-  class EmbulkClient
-    def generate_config(bq_config)
-      Samidare::EmbulkUtility::ConfigGenerator.new.generate_config(database_configs, bq_config)
-    end
-
-    def run(bq_config, target_table_names = [], retry_max = 0)
-      error_tables = run_and_retry(bq_config, target_table_names, retry_max, 0)
-      # return batch status(true: all tables success)
-      error_tables.size == 0
-    end
-
-    private
-    def run_and_retry(bq_config, target_table_names = [], retry_max, retry_count)
-      error_tables = Samidare::Embulk.new.run(
-        database_configs,
-        table_configs,
-        bq_config,
-        target_table_names)
-      if error_tables.size > 0 && retry_count < retry_max
-        puts "------------------------------------"
-        puts "retry start -> #{retry_count + 1} time"
-        puts "------------------------------------"
-        error_tables = run_and_retry(bq_config, error_tables, retry_max, retry_count + 1)
-      end
-      error_tables
-    end
-
-    def database_configs
-      @database_configs ||= YAML.load_file('database.yml')
-    end
-
-    def table_configs
-      @table_configs ||= Samidare::MySQL::TableConfig.generate_table_configs
-    end
-  end
-end
+require 'samidare/version'
+require 'samidare/embulk_utility'
+require 'samidare/embulk'
+require 'samidare/mysql'
+
+module Samidare
+  class EmbulkClient
+    def generate_config(bq_config)
+      Samidare::EmbulkUtility::ConfigGenerator.new.generate_config(database_configs, bq_config)
+    end
+
+    def run(bq_config, target_table_names = [], retry_max = 0)
+      error_tables = run_and_retry(bq_config, target_table_names, retry_max, 0)
+      # return batch status(true: all tables success)
+      error_tables.size == 0
+    end
+
+    private
+    def run_and_retry(bq_config, target_table_names = [], retry_max, retry_count)
+      error_tables = Samidare::Embulk.new.run(
+        database_configs,
+        table_configs,
+        bq_config,
+        target_table_names)
+      if error_tables.size > 0 && retry_count < retry_max
+        puts "------------------------------------"
+        puts "retry start -> #{retry_count + 1} time"
+        puts "------------------------------------"
+        error_tables = run_and_retry(bq_config, error_tables, retry_max, retry_count + 1)
+      end
+      error_tables
+    end
+
+    def database_configs
+      @database_configs ||= YAML.load_file('database.yml')
+    end
+
+    def table_configs
+      @table_configs ||= Samidare::MySQL::TableConfig.generate_table_configs
+    end
+  end
+end

data/lib/samidare/bigquery_utility.rb

@@ -1,85 +1,85 @@
-require 'json'
-require 'erb'
-require 'big_query'
-require 'unindent'
-require 'date'
-
-module Samidare
-  class BigQueryUtility
-    CONTENTS = <<-EOS.unindent
-    in:
-      type: mysql
-      user: <%= user %>
-      password: <%= password %>
-      database: <%= database %>
-      host: <%= host %>
-      query: |
-        <%= query %>
-    out:
-      type: bigquery
-      project: <%= project %>
-      p12_keyfile_path: <%= p12_keyfile_path %>
-      service_account_email: <%= service_account_email %>
-      dataset: <%= dataset %>
-      table: <%= table_name %>
-      schema_path: <%= schema_path %>
-      auto_create_table: 1
-      path_prefix: <%= path_prefix %>
-      source_format: NEWLINE_DELIMITED_JSON
-      file_ext: .json.gz
-      delete_from_local_when_job_end: 1
-      formatter:
-        type: jsonl
-      encoders:
-      - {type: gzip}
-    EOS
-
-    def initialize(config)
-      @config = config.dup
-      @current_date = Date.today
-    end
-
-    def self.generate_schema(columns)
-      json_body = columns.map { |column| column.to_json }.join(",\n")
-      "[\n" + json_body + "\n]\n"
-    end
-
-    def self.generate_sql(table_config, columns)
-      columns = columns.map { |column| column.converted_value }
-      sql = "SELECT " + columns.join(",")
-      sql << " FROM #{table_config.name}"
-      sql << " WHERE #{table_config.condition}" if table_config.condition
-      sql << "\n"
-      sql
-    end
-
-    def generate_embulk_config(db_name, database_config, table_config, columns)
-      host = database_config['host']
-      user = database_config['username']
-      password = database_config['password']
-      database = database_config['database']
-      query = Samidare::BigQueryUtility.generate_sql(table_config, columns)
-      project = @config['project_id']
-      p12_keyfile_path = @config['key']
-      service_account_email = @config['service_email']
-      dataset = database_config['bq_dataset']
-      table_name = actual_table_name(table_config.name, database_config['daily_snapshot'] || table_config.daily_snapshot)
-      schema_path = "#{@config['schema_dir']}/#{db_name}/#{table_config.name}.json"
-      path_prefix = "/var/tmp/embulk_#{db_name}_#{table_config.name}"
-
-      ERB.new(CONTENTS).result(binding)
-    end
-
-    def delete_table(dataset, table_name)
-      @config['dataset'] = dataset
-
-      bq = BigQuery::Client.new(@config)
-      bq.delete_table(table_name)
-    end
-
-    def actual_table_name(table_name, daily_snapshot)
-      return table_name unless daily_snapshot
-      table_name + @current_date.strftime('%Y%m%d')
-    end
-  end
-end
+require 'json'
+require 'erb'
+require 'big_query'
+require 'unindent'
+require 'date'
+
+module Samidare
+  class BigQueryUtility
+    CONTENTS = <<-EOS.unindent
+    in:
+      type: mysql
+      user: <%= user %>
+      password: <%= password %>
+      database: <%= database %>
+      host: <%= host %>
+      query: |
+        <%= query %>
+    out:
+      type: bigquery
+      project: <%= project %>
+      p12_keyfile: <%= p12_keyfile %>
+      service_account_email: <%= service_account_email %>
+      dataset: <%= dataset %>
+      table: <%= table_name %>
+      schema_path: <%= schema_path %>
+      auto_create_table: 1
+      path_prefix: <%= path_prefix %>
+      source_format: NEWLINE_DELIMITED_JSON
+      file_ext: .json.gz
+      delete_from_local_when_job_end: 1
+      formatter:
+        type: jsonl
+      encoders:
+      - {type: gzip}
+    EOS
+
+    def initialize(config)
+      @config = config.dup
+      @current_date = Date.today
+    end
+
+    def self.generate_schema(columns)
+      json_body = columns.map { |column| column.to_json }.join(",\n")
+      "[\n" + json_body + "\n]\n"
+    end
+
+    def self.generate_sql(table_config, columns)
+      columns = columns.map { |column| column.converted_value }
+      sql = "SELECT " + columns.join(",")
+      sql << " FROM #{table_config.name}"
+      sql << " WHERE #{table_config.condition}" if table_config.condition
+      sql << "\n"
+      sql
+    end
+
+    def generate_embulk_config(db_name, database_config, table_config, columns)
+      host = database_config['host']
+      user = database_config['username']
+      password = database_config['password']
+      database = database_config['database']
+      query = Samidare::BigQueryUtility.generate_sql(table_config, columns)
+      project = @config['project_id']
+      p12_keyfile = @config['key']
+      service_account_email = @config['service_email']
+      dataset = database_config['bq_dataset']
+      table_name = actual_table_name(table_config.name, database_config['daily_snapshot'] || table_config.daily_snapshot)
+      schema_path = "#{@config['schema_dir']}/#{db_name}/#{table_config.name}.json"
+      path_prefix = "/var/tmp/embulk_#{db_name}_#{table_config.name}"
+
+      ERB.new(CONTENTS).result(binding)
+    end
+
+    def delete_table(dataset, table_name)
+      @config['dataset'] = dataset
+
+      bq = BigQuery::Client.new(@config)
+      bq.delete_table(table_name)
+    end
+
+    def actual_table_name(table_name, daily_snapshot)
+      return table_name unless daily_snapshot
+      table_name + @current_date.strftime('%Y%m%d')
+    end
+  end
+end