trinamo 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a9f5ffc075b447acba09cc09f5c5422858c887cf
-  data.tar.gz: e5d3de4c641e481365f9ef0bf3d80b20dd2356b2
+  metadata.gz: aaf548285a2c17c8ef8913b70e42058c5dd7d1d2
+  data.tar.gz: 223310522802f3482caa2b46556c068a4f1cd9a4
 SHA512:
-  metadata.gz: e91bcd9fd3b6883fbd039ae1c611e07d33be735cdb3ca56169b579230aad5a54084823eea2624aae21f9090368a1f3f2def0e7563d0b611016bfff9a7d1e0621
-  data.tar.gz: 2b4cad56a5dbda92a7945765b8adb897ea6bf7d77556225dc9fa59a64e6c8529f705f047dbb3e36d05c5d9c33403c7288c68d9703e79faca81a408140ac4d91c
+  metadata.gz: 6ef96ddbe9510141e62835a003e9cbfa03ea518e59fd348ac257bd48b578ef1a0f7e77b6947d50f4cb77e0cd79b40b51e00fc6690fcf2d351cf2bc99ff6412af
+  data.tar.gz: 58e45dcfe9dd86704894282028a5a6f5666f758f0d76052cd52e3e628acfa73c7d90fd55909e51b7941727452a7391b70d78dcf6c2b9b0be7145926d12a02f9c

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2016 cignoir
+Copyright (c) 2016 Shulla Cignoir
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,8 +1,12 @@
 # Trinamo
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/trinamo`. To experiment with that code, run `bin/console` for an interactive prompt.
+Trinamo generates DDL for Hive from YAML
+to mount tables of DynamoDB, S3 and local HDFS.
 
-TODO: Delete this and the text above, and describe your gem
+```
+Notice:
+This is experimental stuff! Do not use in production.
+```
 
 ## Installation
 
@@ -22,17 +26,117 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
+### Create a DDL template
+
+* RUN:
+```ruby
+Trinamo::Converter.generate_template(out_file_path = 'ddl.yml')
+```
+
+* OUTPUT:
+```yaml
+dynamo_read_percent: 0.75
+tables:
+  - name: comments
+    s3_location: s3://path/to/s3/table/location
+    s3_partition:
+      - name: date
+        type: string
+    hash_key:
+      - name: user_id
+        type: bigint
+    range_key:
+      - name: comment_id
+        type: bigint
+    attributes:
+      - name: title
+        type: string
+      - name: content
+        type: string
+      - name: rate
+        type: double
+  - name: authors
+    hash_key:
+      - name: author_id
+        type: bigint
+    attributes:
+      - name: name
+        type: string
+```
+
+### Create a mapper for DynamoDB
+
+* RUN:
+```ruby
+Trinamo::Converter.load('ddl.yml', :dynamodb).convert
+```
+
+* OUTPUT:
+```hql
+SET dynamodb.throughput.read.percent = 0.75;
+SET hive.exec.compress.output=true;
+SET io.seqfile.compression.type=BLOCK;
+SET mapred.output.compression.codec = com.hadoop.compression.lzo.LzoCodec;
+
+-- comments_ddb
+CREATE EXTERNAL TABLE comments_ddb (
+  user_id BIGINT,comment_id BIGINT,title STRING,content STRING,rate DOUBLE
+)
+STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
+TBLPROPERTIES (
+  'dynamodb.table.name' = 'comments',
+  'dynamodb.column.mapping' = 'user_id:user_id,comment_id:comment_id,title:title,content:content,rate:rate'
+);
+
+-- authors_ddb
+CREATE EXTERNAL TABLE authors_ddb (
+  author_id BIGINT,name STRING
+)
+STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
+TBLPROPERTIES (
+  'dynamodb.table.name' = 'authors',
+  'dynamodb.column.mapping' = 'author_id:author_id,name:name'
+);
+```
+
+### Create a mapper for S3
+* RUN:
+```ruby
+Trinamo::Converter.load('ddl.yml', :s3).convert
+```
 
-## Development
+* OUTPUT:
+```hql
+-- comments_s3
+CREATE EXTERNAL TABLE comments_s3 (
+  user_id BIGINT,comment_id BIGINT,title STRING,content STRING,rate DOUBLE
+) PARTITIONED BY (date STRING)
+ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'
+LOCATION 's3://path/to/s3/table/location';
+```
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+### Create a mapper for HDFS local
+* RUN:
+```ruby
+Trinamo::Converter.load('ddl.yml', :hdfs).convert
+```
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+* OUTPUT:
+```hql
+-- comments_hdfs
+CREATE TABLE comments_hdfs (
+  user_id BIGINT,comment_id BIGINT,title STRING,content STRING,rate DOUBLE
+);
+
+-- authors_hdfs
+CREATE TABLE authors_hdfs (
+  author_id BIGINT,name STRING
+);
+```
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/trinamo. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at https://github.com/cignoir/trinamo.
 
 
 ## License

data/lib/trinamo/converter.rb

@@ -0,0 +1,54 @@
+require 'unindent'
+require_relative './converter/dynamodb_converter'
+require_relative './converter/hdfs_converter'
+require_relative './converter/s3_converter'
+
+module Trinamo
+  class Converter
+    class << self
+      def load(ddl_yaml_path, format)
+        case format
+          when :hdfs then HdfsConverter.new(ddl_yaml_path)
+          when :s3 then S3Converter.new(ddl_yaml_path)
+          when :dynamodb then DynamodbConverter.new(ddl_yaml_path)
+          else raise "[ERROR] Unknown format: #{format}" unless [:dynamodb, :hdfs, :s3].include(format)
+        end
+      end
+
+      def generate_template(out_file_path = nil)
+        template = <<-TEMPLATE.unindent
+          dynamo_read_percent: 0.75
+          tables:
+            - name: comments
+              s3_location: s3://path/to/s3/table/location
+              s3_partition:
+                - name: date
+                  type: string
+              hash_key:
+                - name: user_id
+                  type: bigint
+              range_key:
+                - name: comment_id
+                  type: bigint
+              attributes:
+                - name: title
+                  type: string
+                - name: content
+                  type: string
+                - name: rate
+                  type: double
+            - name: authors
+              hash_key:
+                - name: author_id
+                  type: bigint
+              attributes:
+                - name: name
+                  type: string
+        TEMPLATE
+
+        File.binwrite(out_file_path, template) if out_file_path
+        template
+      end
+    end
+  end
+end

data/lib/trinamo/converter/base_converter.rb

@@ -0,0 +1,13 @@
+require 'yaml'
+require 'active_support/core_ext/hash'
+
+module Trinamo
+  class BaseConverter
+    attr_accessor :ddl_yaml_path, :ddl
+
+    def initialize(ddl_yaml_path)
+      @ddl_yaml_path = ddl_yaml_path
+      @ddl = YAML.load_file(ddl_yaml_path).deep_symbolize_keys
+    end
+  end
+end

data/lib/trinamo/converter/dynamodb_converter.rb

@@ -0,0 +1,33 @@
+require_relative './base_converter'
+
+module Trinamo
+  class DynamodbConverter < BaseConverter
+    def convert
+      read_percent = @ddl[:dynamo_read_percent] ? @ddl[:dynamo_read_percent] : 0.5
+
+      ddl_header = <<-DDL.unindent
+        SET dynamodb.throughput.read.percent = #{read_percent};
+        SET hive.exec.compress.output=true;
+        SET io.seqfile.compression.type=BLOCK;
+        SET mapred.output.compression.codec = com.hadoop.compression.lzo.LzoCodec;
+      DDL
+
+      ddl_body = @ddl[:tables].map do |h|
+        fields = ([h[:hash_key]] + [h[:range_key]] + [h[:attributes]]).flatten.compact
+        <<-DDL.unindent
+          -- #{h[:name]}_ddb
+          CREATE EXTERNAL TABLE #{h[:name]}_ddb (
+            #{fields.map { |attr| "#{attr[:name]} #{attr[:type].upcase}" }.join(',')}
+          )
+          STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
+          TBLPROPERTIES (
+            'dynamodb.table.name' = '#{h[:name]}',
+            'dynamodb.column.mapping' = '#{fields.map { |attr| "#{attr[:name]}:#{attr[:name]}" }.join(',') }'
+          );
+        DDL
+      end
+
+      ([ddl_header] + ddl_body).join("\n")
+    end
+  end
+end

data/lib/trinamo/converter/hdfs_converter.rb

@@ -0,0 +1,19 @@
+require_relative './base_converter'
+
+module Trinamo
+  class HdfsConverter < BaseConverter
+    def convert
+      ddl_body = @ddl[:tables].map do |h|
+        fields = ([h[:hash_key]] + [h[:range_key]] + [h[:attributes]]).flatten.compact
+        <<-DDL.unindent
+          -- #{h[:name]}_hdfs
+          CREATE TABLE #{h[:name]}_hdfs (
+            #{fields.map { |attr| "#{attr[:name]} #{attr[:type].upcase}" }.join(',')}
+          );
+        DDL
+      end
+
+      ddl_body.join("\n")
+    end
+  end
+end

data/lib/trinamo/converter/s3_converter.rb

@@ -0,0 +1,27 @@
+require_relative './base_converter'
+
+module Trinamo
+  class S3Converter < BaseConverter
+    def convert
+      ddl_body = @ddl[:tables].map do |h|
+        if h[:s3_location]
+          fields = ([h[:hash_key]] + [h[:range_key]] + [h[:attributes]]).flatten.compact
+          partitioned_by = h[:s3_partition] ? "PARTITIONED BY (#{h[:s3_partition].map { |attr| "#{attr[:name]} #{attr[:type].upcase}" }.join(',')})" : ''
+          <<-DDL.unindent
+            -- #{h[:name]}_s3
+            CREATE EXTERNAL TABLE #{h[:name]}_s3 (
+              #{fields.map { |attr| "#{attr[:name]} #{attr[:type].upcase}" }.join(',')}
+            ) #{partitioned_by}
+            ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'
+            LOCATION '#{h[:s3_location]}';
+          DDL
+        else
+          STDERR.puts "[ERROR] The location of #{h[:name]}_s3 is not found"
+          nil
+        end
+      end
+
+      ddl_body.compact.join("\n")
+    end
+  end
+end

data/lib/trinamo/version.rb

@@ -1,3 +1,3 @@
 module Trinamo
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/trinamo.gemspec

@@ -9,7 +9,7 @@ Gem::Specification.new do |spec|
   spec.authors       = ["cignoir"]
   spec.email         = ["cignoir@gmail.com"]
 
-  spec.summary       = %q{utilities for aws-dynamodb}
+  spec.summary       = %q{DDL Generator for Hive from YAML}
   spec.description   = %q{}
   spec.homepage      = "https://github.com/cignoir/trinamo"
   spec.license       = "MIT"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: trinamo
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - cignoir
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-06-08 00:00:00.000000000 Z
+date: 2016-06-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -88,14 +88,8 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".gitignore"
-- ".idea/encodings.xml"
-- ".idea/misc.xml"
-- ".idea/modules.xml"
-- ".idea/trinamo.iml"
-- ".idea/workspace.xml"
 - ".rspec"
 - ".travis.yml"
-- CODE_OF_CONDUCT.md
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -103,6 +97,11 @@ files:
 - bin/console
 - bin/setup
 - lib/trinamo.rb
+- lib/trinamo/converter.rb
+- lib/trinamo/converter/base_converter.rb
+- lib/trinamo/converter/dynamodb_converter.rb
+- lib/trinamo/converter/hdfs_converter.rb
+- lib/trinamo/converter/s3_converter.rb
 - lib/trinamo/hive_type.rb
 - lib/trinamo/version.rb
 - trinamo.gemspec
@@ -129,5 +128,6 @@ rubyforge_project:
 rubygems_version: 2.4.5.1
 signing_key: 
 specification_version: 4
-summary: utilities for aws-dynamodb
+summary: DDL Generator for Hive from YAML
 test_files: []
+has_rdoc: 

data/.idea/encodings.xml

@@ -1,6 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project version="4">
-  <component name="Encoding">
-    <file url="PROJECT" charset="UTF-8" />
-  </component>
-</project>

data/.idea/misc.xml

@@ -1,14 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project version="4">
-  <component name="ProjectLevelVcsManager" settingsEditedManually="false">
-    <OptionsSetting value="true" id="Add" />
-    <OptionsSetting value="true" id="Remove" />
-    <OptionsSetting value="true" id="Checkout" />
-    <OptionsSetting value="true" id="Update" />
-    <OptionsSetting value="true" id="Status" />
-    <OptionsSetting value="true" id="Edit" />
-    <ConfirmationsSetting value="0" id="Add" />
-    <ConfirmationsSetting value="0" id="Remove" />
-  </component>
-  <component name="ProjectRootManager" version="2" project-jdk-name="ruby-2.1.5-p273" project-jdk-type="RUBY_SDK" />
-</project>

data/.idea/modules.xml

@@ -1,8 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project version="4">
-  <component name="ProjectModuleManager">
-    <modules>
-      <module fileurl="file://$PROJECT_DIR$/.idea/trinamo.iml" filepath="$PROJECT_DIR$/.idea/trinamo.iml" />
-    </modules>
-  </component>
-</project>

data/.idea/trinamo.iml

@@ -1,52 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module type="RUBY_MODULE" version="4">
-  <component name="FacetManager">
-    <facet type="gem" name="New Gem">
-      <configuration>
-        <option name="GEM_APP_ROOT_PATH" value="$MODULE_DIR$" />
-        <option name="GEM_APP_TEST_PATH" value="$MODULE_DIR$/test" />
-        <option name="GEM_APP_LIB_PATH" value="$MODULE_DIR$/lib" />
-      </configuration>
-    </facet>
-  </component>
-  <component name="ModuleRunConfigurationManager">
-    <configuration default="false" name="test.rb" type="RubyRunConfigurationType" factoryName="Ruby">
-      <module name="trinamo" />
-      <RUBY_RUN_CONFIG NAME="RUBY_ARGS" VALUE="-e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift)" />
-      <RUBY_RUN_CONFIG NAME="WORK DIR" VALUE="" />
-      <RUBY_RUN_CONFIG NAME="SHOULD_USE_SDK" VALUE="false" />
-      <RUBY_RUN_CONFIG NAME="ALTERN_SDK_NAME" VALUE="" />
-      <RUBY_RUN_CONFIG NAME="myPassParentEnvs" VALUE="true" />
-      <envs />
-      <EXTENSION ID="BundlerRunConfigurationExtension" bundleExecEnabled="true" />
-      <EXTENSION ID="JRubyRunConfigurationExtension" NailgunExecEnabled="false" />
-      <EXTENSION ID="RubyCoverageRunConfigurationExtension" enabled="false" sample_coverage="true" track_test_folders="true" runner="rcov" />
-      <RUBY_RUN_CONFIG NAME="SCRIPT_PATH" VALUE="$MODULE_DIR$/test.rb" />
-      <RUBY_RUN_CONFIG NAME="SCRIPT_ARGS" VALUE="" />
-      <method />
-    </configuration>
-  </component>
-  <component name="NewModuleRootManager">
-    <content url="file://$MODULE_DIR$">
-      <sourceFolder url="file://$MODULE_DIR$/test" isTestSource="true" />
-      <excludeFolder url="file://$MODULE_DIR$/.bundle" />
-      <excludeFolder url="file://$MODULE_DIR$/vendor/bundle" />
-    </content>
-    <orderEntry type="jdk" jdkName="ruby-2.2.4-p230" jdkType="RUBY_SDK" />
-    <orderEntry type="sourceFolder" forTests="false" />
-    <orderEntry type="library" scope="PROVIDED" name="activesupport (v4.2.6, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="diff-lcs (v1.2.5, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="i18n (v0.7.0, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="json (v1.8.3, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="minitest (v5.9.0, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="rake (v10.5.0, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="rspec (v3.4.0, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="rspec-core (v3.4.4, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="rspec-expectations (v3.4.0, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="rspec-mocks (v3.4.1, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="rspec-support (v3.4.1, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="thread_safe (v0.3.5, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="tzinfo (v1.2.2, ruby-2.2.4-p230) [gem]" level="application" />
-    <orderEntry type="library" scope="PROVIDED" name="unindent (v1.0, ruby-2.2.4-p230) [gem]" level="application" />
-  </component>
-</module>

data/CODE_OF_CONDUCT.md

@@ -1,49 +0,0 @@
-# Contributor Code of Conduct
-
-As contributors and maintainers of this project, and in the interest of
-fostering an open and welcoming community, we pledge to respect all people who
-contribute through reporting issues, posting feature requests, updating
-documentation, submitting pull requests or patches, and other activities.
-
-We are committed to making participation in this project a harassment-free
-experience for everyone, regardless of level of experience, gender, gender
-identity and expression, sexual orientation, disability, personal appearance,
-body size, race, ethnicity, age, religion, or nationality.
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery
-* Personal attacks
-* Trolling or insulting/derogatory comments
-* Public or private harassment
-* Publishing other's private information, such as physical or electronic
-  addresses, without explicit permission
-* Other unethical or unprofessional conduct
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-By adopting this Code of Conduct, project maintainers commit themselves to
-fairly and consistently applying these principles to every aspect of managing
-this project. Project maintainers who do not follow or enforce the Code of
-Conduct may be permanently removed from the project team.
-
-This code of conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community.
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting a project maintainer at cignoir@gmail.com. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. Maintainers are
-obligated to maintain confidentiality with regard to the reporter of an
-incident.
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
-version 1.3.0, available at
-[http://contributor-covenant.org/version/1/3/0/][version]
-
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/3/0/