RubyGems - embulk-output-elasticsearch5 - Versions diffs - 0.1.0 - Mend

embulk-output-elasticsearch5 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (19) hide show

checksums.yaml +7 -0
data/.gitignore +7 -0
data/CHANGELOG.md +51 -0
data/README.md +117 -0
data/build.gradle +110 -0
data/config/checkstyle/checkstyle.xml +130 -0
data/config/checkstyle/default.xml +110 -0
data/gradle/wrapper/gradle-wrapper.jar +0 -0
data/gradle/wrapper/gradle-wrapper.properties +6 -0
data/gradle/wrapper/wrapper/gradle-wrapper.jar +0 -0
data/gradle/wrapper/wrapper/gradle-wrapper.properties +6 -0
data/gradlew +160 -0
data/gradlew.bat +90 -0
data/lib/embulk/output/elasticsearch5.rb +3 -0
data/settings.gradle +1 -0
data/src/main/java/org/embulk/output/elasticsearch/ElasticsearchOutputPlugin.java +630 -0
data/src/test/java/org/embulk/output/elasticsearch/TestElasticsearchOutputPlugin.java +482 -0
data/src/test/resources/sample_01.csv +5 -0
metadata +153 -0

checksums.yaml ADDED Viewed

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: fb678c687d200121d03d61d7d850f131d8daf48b
+  data.tar.gz: 2b514fcefd2c994d5685ccf335109ad5b3c0f77d
+SHA512:
+  metadata.gz: 623a488e51fc2035bfa1c53cf276478e1f02a749b9c34be40f6fe232f2f294e94a2f9bd6e8a472bf3f682dd7b38100206dc72851d3bcbdf62485ac3d970691c4
+  data.tar.gz: 43742e43ef5db38d4935fa3e9480322992fed76c54cff9fc7548b7c64c5a30ba43e14288a890b9f19c4f12bf7e2910eb20a1176d84aebc5fd2d26a287f8a6b00

data/.gitignore ADDED Viewed

@@ -0,0 +1,7 @@
+*~
+*.iml
+.idea
+build/
+/classpath/
+/.gradle
+/pkg/

data/CHANGELOG.md ADDED Viewed

@@ -0,0 +1,51 @@
+## 0.3.1 - 2016-06-21
+* [maintenance] Update Elasticsearch client to 2.3.3 [#25](https://github.com/muga/embulk-output-elasticsearch/pull/25)
+## 0.3.0 - 2016-02-22
+* [maintenance] Upgrade Embulk v08 [#21](https://github.com/muga/embulk-output-elasticsearch/pull/21)
+## 0.2.1 - 2016-02-05
+* [maintenance] Fix bug. Force to fail jobs if nodes down while executing [#19](https://github.com/muga/embulk-output-elasticsearch/pull/19)
+## 0.2.0 - 2016-01-26
+* [new feature] Support Elasticsearch 2.x [#12](https://github.com/muga/embulk-output-elasticsearch/pull/12)
+* [new feature] Added replace mode [#15](https://github.com/muga/embulk-output-elasticsearch/pull/15)
+* [maintenance] Fix id param's behavior [#14](https://github.com/muga/embulk-output-elasticsearch/pull/14)
+* [maintenance] Added unit tests [#17](https://github.com/muga/embulk-output-elasticsearch/pull/17)
+* [maintenance] Upgraded Embulk to v0.7.7
+## 0.1.8 - 2015-08-19
+* [maintenance] Upgraded Embulk to v0.7.0
+* [maintenance] Upgraded Elasticsearch to v1.5.2
+## 0.1.7 - 2015-05-09
+* [maintenance] Fixed handling null value [#10](https://github.com/muga/embulk-output-elasticsearch/pull/10)
+## 0.1.6 - 2015-04-14
+* [new feature] Added bulk_size parameter [#8](https://github.com/muga/embulk-output-elasticsearch/pull/8)
+## 0.1.5 - 2015-03-26
+* [new feature] Added cluster_name parameter [#7](https://github.com/muga/embulk-output-elasticsearch/pull/7)
+## 0.1.4 - 2015-03-19
+* [maintenance] Fixed parameter names index_name to index, doc_id_column to id. [#5](https://github.com/muga/embulk-output-elasticsearch/pull/5)
+* [maintenance] Fixed typo at parameter [#6](https://github.com/muga/embulk-output-elasticsearch/pull/6)
+## 0.1.3 - 2015-02-25
+* [new feature] Supported timestamp column [#4](https://github.com/muga/embulk-output-elasticsearch/pull/4)
+## 0.1.2 - 2015-02-24
+## 0.1.1 - 2015-02-16
+## 0.1.0 - 2015-02-16

data/README.md ADDED Viewed

@@ -0,0 +1,117 @@
+# Elasticsearch output plugin for Embulk
+**Notice** This plugin doesn't support [Amazon(AWS) Elasticsearch Service](https://aws.amazon.com/elasticsearch-service/).
+Plugin uses [Transport Client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.0/transport-client.html) but AWS Elasticsearch doesn't support this method.
+> The service supports HTTP on port 80, but does not support TCP transport.
+- *[Amazon Elasticsearch Service Limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html)*
+## Overview
+* **Plugin type**: output
+* **Rollback supported**: no
+* **Resume supported**: no
+* **Cleanup supported**: no
+## Configuration
+- **mode**: "insert" or "replace". See below(string, optional, default is insert)
+- **nodes**: list of nodes. nodes are pairs of host and port (list, required)
+- **cluster_name**: name of the cluster (string, default is "elasticsearch")
+- **index**: index name (string, required)
+- **index_type**: index type (string, required)
+- **id**: document id column (string, default is null)
+- **bulk_actions**: Sets when to flush a new bulk request based on the number of actions currently added. (int, default is 1000)
+- **bulk_size**: Sets when to flush a new bulk request based on the size of actions currently added. (long, default is 5242880)
+- **concurrent_requests**: concurrent_requests (int, default is 5)
+### Modes
+#### insert:
+default.
+This mode writes data to existing index.
+#### replace:
+1. Create new temporary index
+2. Insert data into the new index
+3. replace the alias with the new index. If alias doesn't exists, plugin will create new alias.
+4. Delete existing (old) index if exists
+Index should not exists with the same name as the alias
+```yaml
+out:
+  type: elasticsearch
+  mode: replace
+  nodes:
+  - {host: localhost, port: 9300}
+  index: <alias name> # plugin generates index name like <index>_%Y%m%d-%H%M%S
+  index_type: <index type>
+```
+## Example
+```yaml
+out:
+  type: elasticsearch
+  mode: insert
+  nodes:
+  - {host: localhost, port: 9300}
+  index: <index name>
+  index_type: <index type>
+```
+## Build
+```
+$ ./gradlew gem  # -t to watch change of files and rebuild continuously
+```
+## Test
+```
+$ ./gradlew test  # -t to watch change of files and rebuild continuously
+```
+To run unit tests, we need to configure the following environment variables.
+When environment variables are not set, skip almost test cases.
+```
+ES_HOST
+ES_PORT(optional, if needed, default: 9300)
+ES_INDEX
+ES_INDEX_TYPE
+```
+If you're using Mac OS X El Capitan and GUI Applications(IDE), like as follows.
+```
+$ vi ~/Library/LaunchAgents/environment.plist
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>Label</key>
+  <string>my.startup</string>
+  <key>ProgramArguments</key>
+  <array>
+    <string>sh</string>
+    <string>-c</string>
+    <string>
+      launchctl setenv ES_HOST example.com
+      launchctl setenv ES_PORT 9300
+      launchctl setenv ES_INDEX embulk
+      launchctl setenv ES_INDEX_TYPE embulk
+    </string>
+  </array>
+  <key>RunAtLoad</key>
+  <true/>
+</dict>
+</plist>
+$ launchctl load ~/Library/LaunchAgents/environment.plist
+$ launchctl getenv ES_INDEX //try to get value.
+Then start your applications.
+```

data/build.gradle ADDED Viewed

@@ -0,0 +1,110 @@
+plugins {
+    id "com.jfrog.bintray" version "1.1"
+    id "com.github.jruby-gradle.base" version "0.1.5"
+    id "java"
+    id "jacoco"
+    id "checkstyle"
+}
+import com.github.jrubygradle.JRubyExec
+repositories {
+    mavenCentral()
+    jcenter()
+    mavenLocal()
+}
+configurations {
+    provided
+}
+version = "0.1.0"
+compileJava.options.encoding = 'UTF-8' // source encoding
+sourceCompatibility = 1.7
+targetCompatibility = 1.7
+dependencies {
+    compile  "org.embulk:embulk-core:0.8.5"
+    provided  "org.embulk:embulk-core:0.8.5"
+    compile "org.elasticsearch:elasticsearch:5.1.1"
+    compile "org.elasticsearch.client:transport:5.1.1"
+    testCompile "junit:junit:4.+"
+    testCompile "org.embulk:embulk-core:0.8.5:tests"
+    testCompile "org.embulk:embulk-standards:0.8.5"
+}
+task classpath(type: Copy, dependsOn: ["jar"]) {
+    doFirst { file("classpath").deleteDir() }
+    from (configurations.runtime - configurations.provided + files(jar.archivePath))
+    into "classpath"
+}
+clean { delete 'classpath' }
+checkstyle {
+    configFile = file("${project.rootDir}/config/checkstyle/checkstyle.xml")
+    toolVersion = '6.14.1'
+}
+checkstyleMain {
+    configFile = file("${project.rootDir}/config/checkstyle/default.xml")
+    ignoreFailures = true
+}
+checkstyleTest {
+    configFile = file("${project.rootDir}/config/checkstyle/default.xml")
+    ignoreFailures = true
+}
+task checkstyle(type: Checkstyle) {
+    classpath = sourceSets.main.output + sourceSets.test.output
+    source = sourceSets.main.allJava + sourceSets.test.allJava
+}
+//task copyDependencies(type:Copy) {
+//    new File("$buildDir/libs/dependencies").mkdirs()
+//    into "$buildDir/libs/dependencies" from configurations.runtime
+//}
+task gem(type: JRubyExec, dependsOn: ["build", "gemspec", "classpath"]) {
+    jrubyArgs "-rrubygems/gem_runner", "-eGem::GemRunner.new.run(ARGV)", "build"
+    script "build/gemspec"
+    doLast { ant.move(file: "${project.name}-${project.version}.gem", todir: "pkg") }
+}
+task gemPush(type: JRubyExec, dependsOn: ["gem"]) {
+    jrubyArgs "-rrubygems/gem_runner", "-eGem::GemRunner.new.run(ARGV)", "push"
+    script "pkg/${project.name}-${project.version}.gem"
+}
+task "package"(dependsOn: ["gemspec", "classpath"]) << {
+    println "> Build succeeded."
+    println "> You can run embulk with '-L ${file(".").absolutePath}' argument."
+}
+task gemspec << { file("build/gemspec").write($/
+Gem::Specification.new do |spec|
+  spec.name          = "${project.name}"
+  spec.version       = "${project.version}"
+  spec.authors       = ["Muga Nishizawa"]
+  spec.summary       = %[Elasticsearch5 output plugin for Embulk]
+  spec.description   = %[Elasticsearch5 output plugin is an Embulk plugin that loads records to Elasticsearch read by any input plugins. Search the input plugins by "embulk-input" keyword.]
+  spec.email         = ["muga.nishizawa@gmail.com","tristan.slominski@gmail.com"]
+  spec.licenses      = ["Apache 2.0"]
+  spec.homepage      = "https://github.com/tristanls/embulk-output-elasticsearch"
+  spec.files         = `git ls-files`.split("\n") + Dir["classpath/*.jar"]
+  spec.test_files    = spec.files.grep(%r"^(test|spec)/")
+  spec.require_paths = ["lib"]
+  spec.executables   = spec.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  spec.has_rdoc      = false
+  spec.add_development_dependency "bundler", [">= 1.0"]
+  spec.add_development_dependency "rake", [">= 10.0"]
+  spec.add_development_dependency "test-unit", ["~> 3.0.2"]
+end
+/$)
+}
+jacocoTestReport {
+    afterEvaluate {
+        classDirectories = files(classDirectories.files.collect {
+            fileTree(dir: it, exclude: 'org/embulk/output/elasticsearch/ElasticsearchOutputPlugin$1.class')
+        })
+    }
+}

data/config/checkstyle/checkstyle.xml ADDED Viewed

@@ -0,0 +1,130 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE module PUBLIC
+        "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
+        "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
+<module name="Checker">
+    <!-- https://github.com/facebook/presto/blob/master/src/checkstyle/checks.xml -->
+    <module name="FileTabCharacter"/>
+    <module name="NewlineAtEndOfFile">
+        <property name="lineSeparator" value="lf"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\r"/>
+        <property name="message" value="Line contains carriage return"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value=" \n"/>
+        <property name="message" value="Line has trailing whitespace"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\{\n\n"/>
+        <property name="message" value="Blank line after opening brace"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\s*\}"/>
+        <property name="message" value="Blank line before closing brace"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\n"/>
+        <property name="message" value="Multiple consecutive blank lines"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\Z"/>
+        <property name="message" value="Blank line before end of file"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="Preconditions\.checkNotNull"/>
+        <property name="message" value="Use of checkNotNull"/>
+    </module>
+    <module name="TreeWalker">
+        <module name="EmptyBlock">
+            <property name="option" value="text"/>
+            <property name="tokens" value="
+                LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_IF,
+                LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, STATIC_INIT"/>
+        </module>
+        <module name="EmptyStatement"/>
+        <module name="EmptyForInitializerPad"/>
+        <module name="EmptyForIteratorPad">
+            <property name="option" value="space"/>
+        </module>
+        <module name="MethodParamPad">
+            <property name="allowLineBreaks" value="true"/>
+            <property name="option" value="nospace"/>
+        </module>
+        <module name="ParenPad"/>
+        <module name="TypecastParenPad"/>
+        <module name="NeedBraces"/>
+        <module name="LeftCurly">
+            <property name="option" value="nl"/>
+            <property name="tokens" value="CLASS_DEF, CTOR_DEF, INTERFACE_DEF, METHOD_DEF"/>
+        </module>
+        <module name="LeftCurly">
+            <property name="option" value="eol"/>
+            <property name="tokens" value="
+                 LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_FOR,
+                 LITERAL_IF, LITERAL_SWITCH, LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE"/>
+        </module>
+        <module name="RightCurly">
+            <property name="option" value="alone"/>
+        </module>
+        <module name="GenericWhitespace"/>
+        <module name="WhitespaceAfter"/>
+        <module name="NoWhitespaceBefore"/>
+        <module name="UpperEll"/>
+        <module name="DefaultComesLast"/>
+        <module name="ArrayTypeStyle"/>
+        <module name="MultipleVariableDeclarations"/>
+        <module name="ModifierOrder"/>
+        <module name="OneStatementPerLine"/>
+        <module name="StringLiteralEquality"/>
+        <module name="MutableException"/>
+        <module name="EqualsHashCode"/>
+        <module name="InnerAssignment"/>
+        <module name="InterfaceIsType"/>
+        <module name="HideUtilityClassConstructor"/>
+        <module name="MemberName"/>
+        <module name="LocalVariableName"/>
+        <module name="LocalFinalVariableName"/>
+        <module name="TypeName"/>
+        <module name="PackageName"/>
+        <module name="ParameterName"/>
+        <module name="StaticVariableName">
+            <property name="format" value="^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$"/>
+        </module>
+        <module name="ClassTypeParameterName">
+            <property name="format" value="^[A-Z][0-9]?$"/>
+        </module>
+        <module name="MethodTypeParameterName">
+            <property name="format" value="^[A-Z][0-9]?$"/>
+        </module>
+        <module name="AvoidStarImport"/>
+        <module name="RedundantImport"/>
+        <module name="UnusedImports"/>
+        <module name="ImportOrder">
+            <property name="groups" value="*,javax,java"/>
+            <property name="separated" value="true"/>
+            <property name="option" value="bottom"/>
+            <property name="sortStaticImportsAlphabetically" value="true"/>
+        </module>
+        <module name="WhitespaceAround">
+            <property name="allowEmptyConstructors" value="true"/>
+            <property name="allowEmptyMethods" value="true"/>
+            <property name="ignoreEnhancedForColon" value="false"/>
+            <property name="tokens" value="
+                ASSIGN, BAND, BAND_ASSIGN, BOR, BOR_ASSIGN, BSR, BSR_ASSIGN,
+                BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN, EQUAL, GE, GT, LAND, LE,
+                LITERAL_ASSERT, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
+                LITERAL_FINALLY, LITERAL_FOR, LITERAL_IF, LITERAL_RETURN,
+                LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE,
+                LOR, LT, MINUS, MINUS_ASSIGN, MOD, MOD_ASSIGN, NOT_EQUAL,
+                PLUS, PLUS_ASSIGN, QUESTION, SL, SLIST, SL_ASSIGN, SR, SR_ASSIGN,
+                STAR, STAR_ASSIGN, TYPE_EXTENSION_AND"/>
+        </module>
+    </module>
+</module>

data/config/checkstyle/default.xml ADDED Viewed

@@ -0,0 +1,110 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE module PUBLIC
+        "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
+        "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
+<!--
+     This is a subset of ./checkstyle.xml which allows some loose styles
+-->
+<module name="Checker">
+    <module name="FileTabCharacter"/>
+    <module name="NewlineAtEndOfFile">
+        <property name="lineSeparator" value="lf"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\r"/>
+        <property name="message" value="Line contains carriage return"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value=" \n"/>
+        <property name="message" value="Line has trailing whitespace"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\n"/>
+        <property name="message" value="Multiple consecutive blank lines"/>
+    </module>
+    <module name="RegexpMultiline">
+        <property name="format" value="\n\n\Z"/>
+        <property name="message" value="Blank line before end of file"/>
+    </module>
+    <module name="TreeWalker">
+        <module name="EmptyBlock">
+            <property name="option" value="text"/>
+            <property name="tokens" value="
+                LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_IF,
+                LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, STATIC_INIT"/>
+        </module>
+        <module name="EmptyStatement"/>
+        <module name="EmptyForInitializerPad"/>
+        <module name="EmptyForIteratorPad">
+            <property name="option" value="space"/>
+        </module>
+        <module name="MethodParamPad">
+            <property name="allowLineBreaks" value="true"/>
+            <property name="option" value="nospace"/>
+        </module>
+        <module name="ParenPad"/>
+        <module name="TypecastParenPad"/>
+        <module name="NeedBraces"/>
+        <module name="LeftCurly">
+            <property name="option" value="nl"/>
+            <property name="tokens" value="CLASS_DEF, CTOR_DEF, INTERFACE_DEF, METHOD_DEF"/>
+        </module>
+        <module name="LeftCurly">
+            <property name="option" value="eol"/>
+            <property name="tokens" value="
+                 LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_FOR,
+                 LITERAL_IF, LITERAL_SWITCH, LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE"/>
+        </module>
+        <module name="RightCurly">
+            <property name="option" value="alone"/>
+        </module>
+        <module name="GenericWhitespace"/>
+        <module name="WhitespaceAfter"/>
+        <module name="NoWhitespaceBefore"/>
+        <module name="UpperEll"/>
+        <module name="DefaultComesLast"/>
+        <module name="ArrayTypeStyle"/>
+        <module name="MultipleVariableDeclarations"/>
+        <module name="ModifierOrder"/>
+        <module name="OneStatementPerLine"/>
+        <module name="StringLiteralEquality"/>
+        <module name="MutableException"/>
+        <module name="EqualsHashCode"/>
+        <module name="InnerAssignment"/>
+        <module name="InterfaceIsType"/>
+        <module name="HideUtilityClassConstructor"/>
+        <module name="MemberName"/>
+        <module name="LocalVariableName"/>
+        <module name="LocalFinalVariableName"/>
+        <module name="TypeName"/>
+        <module name="PackageName"/>
+        <module name="ParameterName"/>
+        <module name="StaticVariableName">
+            <property name="format" value="^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$"/>
+        </module>
+        <module name="ClassTypeParameterName">
+            <property name="format" value="^[A-Z][0-9]?$"/>
+        </module>
+        <module name="MethodTypeParameterName">
+            <property name="format" value="^[A-Z][0-9]?$"/>
+        </module>
+        <module name="WhitespaceAround">
+            <property name="allowEmptyConstructors" value="true"/>
+            <property name="allowEmptyMethods" value="true"/>
+            <property name="ignoreEnhancedForColon" value="false"/>
+            <property name="tokens" value="
+                ASSIGN, BAND, BAND_ASSIGN, BOR, BOR_ASSIGN, BSR, BSR_ASSIGN,
+                BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN, EQUAL, GE, GT, LAND, LE,
+                LITERAL_ASSERT, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
+                LITERAL_FINALLY, LITERAL_FOR, LITERAL_IF, LITERAL_RETURN,
+                LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE,
+                LOR, LT, MINUS, MINUS_ASSIGN, MOD, MOD_ASSIGN, NOT_EQUAL,
+                PLUS, PLUS_ASSIGN, QUESTION, SL, SLIST, SL_ASSIGN, SR, SR_ASSIGN,
+                STAR, STAR_ASSIGN, TYPE_EXTENSION_AND"/>
+        </module>
+    </module>
+</module>