embulk-output-dynamodb 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0c86de91e5c36f167f01b9a3d9510eed81d58527
+  data.tar.gz: 01ca2686938850340d6d6aeb166aa78833ea2603
+SHA512:
+  metadata.gz: 3f3b54201891fb212456b7403dd45bb6a2c7037bb592b67bdba8d51cbf9dacf0097d9e27f8b08c6a4b8280e061f9e981e2c82b97870072b339aaab24b8ed03c2
+  data.tar.gz: 3ff9b88e413a761ff8be4e40abab6c3418a0443fcfb2dedd09e9100b640c03990c5a1dcee82db220d3dc4410a69f2fc3a619873f3606c678eb882f73cf2e7730

data/.gitignore

@@ -0,0 +1,14 @@
+target/
+build/
+pkg/
+*.iml
+*~
+._*
+.idea
+tmp/
+vendor/
+/classpath/
+/.bundle
+.yardoc
+/embulk-*.jar
+/.gradle

data/CHANGELOG.md

@@ -0,0 +1 @@
+## 0.1.0 - 2016-05-13

data/README.md

@@ -0,0 +1,210 @@
+# AWS DynamoDB output plugin for Embulk
+
+[Embulk](http://www.embulk.org/) output plugin to load/insert data into [AWS DynamoDB](http://aws.amazon.com/dynamodb/)
+
+## Overview
+
+* **Plugin type**: output
+* **Load all or nothing**: no
+* **Resume supported**: no
+* **Cleanup supported**: no
+
+## Configuration
+
+### General Parameters
+
+- **mode** (string optional, default:upsert)
+    - **upsert**
+    
+        Creates a new item, or replaces an old item with a new item.
+        If an item that has the same primary key as the new item already exists in the specified table, the new item completely replaces the existing item.
+        This mode uses [BatchWriteItem](http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) and you can set `max_put_items` parameter.
+    
+    - **upsert_with_expression**
+    
+        Edits an existing item's attributes, or adds a new item to the table if it does not already exist.
+        If you have data with following schema, you can set `update_expression` as follows.
+        ```
+        - col1(Number primaryKey)
+        - col2(Number)
+        - col3(String)
+        - col4(Number)
+        - col5(String)
+        ```
+        
+        ```yaml
+        mode: upsert_with_expression
+        # update where col1 has #col1
+        update_expression: "add #col2 :col2 set #col3 = :col3 set #col4 = #col4 - :col4 remove #col5"
+        ```
+
+- **region** (string required) Region of Amazon DynamoDB
+- **auto_create_table** (boolean optional, default: false) [See below]()
+- **table** (string required) Table name of Amazon DynamoDB
+
+    table and option accept [Time#strftime](http://ruby-doc.org/core-2.2.3/Time.html#method-i-strftime) format to construct table name.
+    Table name is formatted at runtime using the local time of the embulk server.
+    
+    For example, with the configuration below, data is inserted into tables table_2015_04, table_2015_05 and so on.
+    ```
+    out:
+      type: dynamodb
+      table: table_%Y_%m
+    ```
+- **write_capacity_units** (int optional) Provisioned write capacity units
+    - **normal** (int optional) value that will be set after execution
+    - **raise** (int optional) value that will be set before execution
+- **read_capacity_units** (int optional) Provisioned read capacity units
+    - **normal** (int optional) value that will be set after execution
+    - **raise** (int optional) value that will be set before execution
+
+        - Up to 30 before execution and down to 5 after execution.
+        ```yaml
+        write_capacity_units:
+        normal: 5
+        raise: 30
+        ```
+
+        - Up to 30 before execution
+        ```yaml
+        write_capacity_units:
+        raise: 30
+        ```
+    
+        - Down to 5 after execution
+        ```yaml
+        write_capacity_units:
+        normal: 5
+        ```
+
+        **NOTICE**: You can decrease the read_capacity_units or write_capacity_units settings for a table, but no more than [4 times per table in a single UTC calendar day](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#d0e53216).
+
+- **max_put_items** (int optional, default:25)
+  Max put items at every batch writing requests when `upsert mode`.
+  A single call to [BatchWriteItem](http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) can write up to 16 MB of data, which can comprise as many as 25 put requests.
+  If you have exceeded data, decrease this value.
+
+### Authentication Parameters
+
+6 authentication methods are supported.
+
+- **basic**
+
+    Use access_key_id and secret_access_key
+    
+    ```yaml
+    auth_method: basic
+    access_key_id: YOUR_ACCESS_KEY_ID
+    secret_access_key: YOUR_SECRET_ACCESS_KEY
+    ```
+
+    - **access_key_id**: AWS access key ID (string, required)
+    
+    - **secret_access_key**: AWS secret access key (string, required)
+
+- **env**
+
+    Read AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY) from environment variables.
+    
+    ```yaml
+    auth_method: env
+    ```
+
+- **instance**
+    
+    Use EC2 instance profile
+    
+    ```yaml
+    auth_method: instance
+    ```
+
+- **profile**
+
+    Read credentials from file. Format of the file is as following, where `[...]` is a name of profile.
+    
+    ```yaml
+    auth_method: profile
+    profile_file: /path/to/profilefile
+    profile_name: PROFILE_NAME
+    ```
+    
+    - **profile_file**: path to a profiles file. (string, default: given by AWS_CREDENTIAL_PROFILES_FILE environment varialbe, or ~/.aws/credentials).
+    
+    - **profile_name**: name of a profile. (string, default: `"default"`)
+    
+    ```
+    [default]
+    aws_access_key_id=YOUR_ACCESS_KEY_ID
+    aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
+    
+    [profile2]
+    ...
+    ```
+
+- **properties**
+
+    Read aws.accessKeyId and aws.secretKey from Java system properties
+    
+    ```yaml
+    auth_method: properties
+    ```
+
+- **session**
+
+    Use temporary-generated access_key_id, secret_access_key and session_token.
+    
+    ```yaml
+    auth_method: session
+    access_key_id: YOUR_ACCESS_KEY_ID
+    secret_access_key: YOUR_SECRET_ACCESS_KEY
+    session_token: SESSION_TOKEN
+    ```
+    
+    - **access_key_id**: AWS access key ID (string, required)
+    
+    - **secret_access_key**: AWS secret access key (string, required)
+    
+    - **session_token**: session token (string, required)
+
+## Example
+
+```yaml
+out:
+  type: dynamodb
+  mode: upsert
+  region: us-west-2
+  auth_method: basic
+  access_key_id: ABCXYZ123ABCXYZ123
+  secret_access_key: ABCXYZ123ABCXYZ123
+  auto_create_table: true
+  table: dynamotable
+  write_capacity_units:
+    normal: 5
+    raise: 20
+  read_capacity_units:
+    normal: 6
+    raise: 30
+```
+
+
+## Build
+
+```
+$ ./gradlew gem  # -t to watch change of files and rebuild continuously
+```
+
+## Test
+
+```
+$ ./gradlew test  # -t to watch change of files and rebuild continuously
+```
+
+To run unit tests, you need to setup DynamoDB on your local environment.
+Please reference the documentation [Running DynamoDB on Your Computer](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html) on AWS Developer Guide.
+```
+$ mkdir /path/to/dynamodb
+$ cd /path/to/dynamodb
+$ wget http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.tar.gz
+$ tar zxvf dynamodb_local_latest.tar.gz
+$ java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb
+```

data/build.gradle

@@ -0,0 +1,92 @@
+plugins {
+    id "com.jfrog.bintray" version "1.1"
+    id "com.github.jruby-gradle.base" version "0.1.5"
+    id "java"
+    id "jacoco"
+    id "checkstyle"
+}
+import com.github.jrubygradle.JRubyExec
+repositories {
+    mavenCentral()
+    jcenter()
+}
+configurations {
+    provided
+}
+
+version = "0.1.0"
+
+sourceCompatibility = 1.7
+targetCompatibility = 1.7
+
+dependencies {
+    compile  "org.embulk:embulk-core:0.8.8"
+    provided "org.embulk:embulk-core:0.8.8"
+    compile "com.amazonaws:aws-java-sdk-dynamodb:1.10.50"
+
+    testCompile "junit:junit:4.+"
+    testCompile "org.embulk:embulk-core:0.8.8:tests"
+    testCompile "org.embulk:embulk-standards:0.8.8"
+}
+
+task classpath(type: Copy, dependsOn: ["jar"]) {
+    doFirst { file("classpath").deleteDir() }
+    from (configurations.runtime - configurations.provided + files(jar.archivePath))
+    into "classpath"
+}
+clean { delete "classpath" }
+
+checkstyle {
+    configFile = file("${project.rootDir}/config/checkstyle/checkstyle.xml")
+    toolVersion = '6.14.1'
+}
+checkstyleMain {
+    configFile = file("${project.rootDir}/config/checkstyle/default.xml")
+    ignoreFailures = true
+}
+checkstyleTest {
+    configFile = file("${project.rootDir}/config/checkstyle/default.xml")
+    ignoreFailures = true
+}
+task checkstyle(type: Checkstyle) {
+    classpath = sourceSets.main.output + sourceSets.test.output
+    source = sourceSets.main.allJava + sourceSets.test.allJava
+}
+
+task gem(type: JRubyExec, dependsOn: ["assemble", "gemspec", "classpath"]) {
+    jrubyArgs "-rrubygems/gem_runner", "-eGem::GemRunner.new.run(ARGV)", "build"
+    script "build/gemspec"
+    doLast { ant.move(file: "${project.name}-${project.version}.gem", todir: "pkg") }
+}
+
+task gemPush(type: JRubyExec, dependsOn: ["gem"]) {
+    jrubyArgs "-rrubygems/gem_runner", "-eGem::GemRunner.new.run(ARGV)", "push"
+    script "pkg/${project.name}-${project.version}.gem"
+}
+
+task "package"(dependsOn: ["gemspec", "classpath"]) << {
+    println "> Build succeeded."
+    println "> You can run embulk with '-L ${file(".").absolutePath}' argument."
+}
+
+task gemspec << { file("build/gemspec").write($/
+Gem::Specification.new do |spec|
+  spec.name          = "${project.name}"
+  spec.version       = "${project.version}"
+  spec.authors       = ["Satoshi Akama"]
+  spec.summary       = %[AWS DynamoDB output plugin for Embulk]
+  spec.description   = %[Dumps records to AWS DynamoDB.]
+  spec.email         = ["satoshiakama@gmail.com"]
+  spec.licenses      = ["Apache-2.0"]
+  # TODO set this: spec.homepage      = "https://github.com/sakama/embulk-output-dynamodb"
+
+  spec.files         = `git ls-files`.split("\n") + Dir["classpath/*.jar"]
+  spec.test_files    = spec.files.grep(%r"^(test|spec)/")
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency 'bundler', ['~> 1.0']
+  spec.add_development_dependency 'rake', ['>= 10.0']
+end
+/$)
+}
+clean { delete "${project.name}.gemspec" }

data/config/checkstyle/checkstyle.xml

@@ -0,0 +1,130 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE module PUBLIC
+        "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
+        "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
+<module name="Checker">
+    <!-- https://github.com/facebook/presto/blob/master/src/checkstyle/checks.xml -->
+    <module name="FileTabCharacter"/>
+    <module name="NewlineAtEndOfFile">
+        <property name="lineSeparator" value="lf"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\r"/>
+        <property name="message" value="Line contains carriage return"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value=" \n"/>
+        <property name="message" value="Line has trailing whitespace"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\{\n\n"/>
+        <property name="message" value="Blank line after opening brace"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\s*\}"/>
+        <property name="message" value="Blank line before closing brace"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\n"/>
+        <property name="message" value="Multiple consecutive blank lines"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\Z"/>
+        <property name="message" value="Blank line before end of file"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="Preconditions\.checkNotNull"/>
+        <property name="message" value="Use of checkNotNull"/>
+    </module>
+
+    <module name="TreeWalker">
+        <module name="EmptyBlock">
+            <property name="option" value="text"/>
+            <property name="tokens" value="
+                LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_IF,
+                LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, STATIC_INIT"/>
+        </module>
+        <module name="EmptyStatement"/>
+        <module name="EmptyForInitializerPad"/>
+        <module name="EmptyForIteratorPad">
+            <property name="option" value="space"/>
+        </module>
+        <module name="MethodParamPad">
+            <property name="allowLineBreaks" value="true"/>
+            <property name="option" value="nospace"/>
+        </module>
+        <module name="ParenPad"/>
+        <module name="TypecastParenPad"/>
+        <module name="NeedBraces"/>
+        <module name="LeftCurly">
+            <property name="option" value="nl"/>
+            <property name="tokens" value="CLASS_DEF, CTOR_DEF, INTERFACE_DEF, METHOD_DEF"/>
+        </module>
+        <module name="LeftCurly">
+            <property name="option" value="eol"/>
+            <property name="tokens" value="
+                 LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_FOR,
+                 LITERAL_IF, LITERAL_SWITCH, LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE"/>
+        </module>
+        <module name="RightCurly">
+            <property name="option" value="alone"/>
+        </module>
+        <module name="GenericWhitespace"/>
+        <module name="WhitespaceAfter"/>
+        <module name="NoWhitespaceBefore"/>
+
+        <module name="UpperEll"/>
+        <module name="DefaultComesLast"/>
+        <module name="ArrayTypeStyle"/>
+        <module name="MultipleVariableDeclarations"/>
+        <module name="ModifierOrder"/>
+        <module name="OneStatementPerLine"/>
+        <module name="StringLiteralEquality"/>
+        <module name="MutableException"/>
+        <module name="EqualsHashCode"/>
+        <module name="InnerAssignment"/>
+        <module name="InterfaceIsType"/>
+        <module name="HideUtilityClassConstructor"/>
+
+        <module name="MemberName"/>
+        <module name="LocalVariableName"/>
+        <module name="LocalFinalVariableName"/>
+        <module name="TypeName"/>
+        <module name="PackageName"/>
+        <module name="ParameterName"/>
+        <module name="StaticVariableName">
+            <property name="format" value="^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$"/>
+        </module>
+        <module name="ClassTypeParameterName">
+            <property name="format" value="^[A-Z][0-9]?$"/>
+        </module>
+        <module name="MethodTypeParameterName">
+            <property name="format" value="^[A-Z][0-9]?$"/>
+        </module>
+
+        <module name="AvoidStarImport"/>
+        <module name="RedundantImport"/>
+        <module name="UnusedImports"/>
+        <module name="ImportOrder">
+            <property name="groups" value="*,javax,java"/>
+            <property name="separated" value="true"/>
+            <property name="option" value="bottom"/>
+            <property name="sortStaticImportsAlphabetically" value="true"/>
+        </module>
+
+        <module name="WhitespaceAround">
+            <property name="allowEmptyConstructors" value="true"/>
+            <property name="allowEmptyMethods" value="true"/>
+            <property name="ignoreEnhancedForColon" value="false"/>
+            <property name="tokens" value="
+                ASSIGN, BAND, BAND_ASSIGN, BOR, BOR_ASSIGN, BSR, BSR_ASSIGN,
+                BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN, EQUAL, GE, GT, LAND, LE,
+                LITERAL_ASSERT, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
+                LITERAL_FINALLY, LITERAL_FOR, LITERAL_IF, LITERAL_RETURN,
+                LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE,
+                LOR, LT, MINUS, MINUS_ASSIGN, MOD, MOD_ASSIGN, NOT_EQUAL,
+                PLUS, PLUS_ASSIGN, QUESTION, SL, SLIST, SL_ASSIGN, SR, SR_ASSIGN,
+                STAR, STAR_ASSIGN, TYPE_EXTENSION_AND"/>
+        </module>
+    </module>
+</module>