embulk-output-elasticsearch 0.1.8 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +39 -0
- data/README.md +77 -1
- data/build.gradle +17 -6
- data/src/main/java/org/embulk/output/elasticsearch/ElasticsearchOutputPlugin.java +182 -39
- data/src/test/java/org/embulk/output/elasticsearch/TestElasticsearchOutputPlugin.java +479 -0
- data/src/test/resources/sample_01.csv +5 -0
- metadata +44 -32
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 17a82e00d22fd47a3f0921903a8b2210c0ef938c
|
|
4
|
+
data.tar.gz: accef942b8a09e8700477fc74883693702eaae0e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7cee36d9a5099ce50c92fdf760777e1a9e6cc960a9d543486d8aad10b0244a97e87e1396e8e653759f16a44ae77a5d9cb66b8be1682221d2b1337b900156c8c9
|
|
7
|
+
data.tar.gz: 75288939178b00e2d612f640c07fee9f0b86b21d1b5d17e4b15c93b2be97cb32e616a042302bf653d5aef09ce94ad790a481d6a952240d3d7700f5c10a29c1d2
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
## 0.2.0 - 2016-01-26
|
|
2
|
+
|
|
3
|
+
* [new feature] Support Elasticsearch 2.x [#12](https://github.com/muga/embulk-output-elasticsearch/pull/12)
|
|
4
|
+
* [new feature] Added replace mode [#15](https://github.com/muga/embulk-output-elasticsearch/pull/15)
|
|
5
|
+
* [maintenance] Fix id param's behavior [#14](https://github.com/muga/embulk-output-elasticsearch/pull/14)
|
|
6
|
+
* [maintenance] Added unit tests [#17](https://github.com/muga/embulk-output-elasticsearch/pull/17)
|
|
7
|
+
* [maintenance] Upgraded Embulk to v0.7.7
|
|
8
|
+
|
|
9
|
+
## 0.1.8 - 2015-08-19
|
|
10
|
+
|
|
11
|
+
* [maintenance] Upgraded Embulk to v0.7.0
|
|
12
|
+
* [maintenance] Upgraded Elasticsearch to v1.5.2
|
|
13
|
+
|
|
14
|
+
## 0.1.7 - 2015-05-09
|
|
15
|
+
|
|
16
|
+
* [maintenance] Fixed handling null value [#10](https://github.com/muga/embulk-output-elasticsearch/pull/10)
|
|
17
|
+
|
|
18
|
+
## 0.1.6 - 2015-04-14
|
|
19
|
+
|
|
20
|
+
* [new feature] Added bulk_size parameter [#8](https://github.com/muga/embulk-output-elasticsearch/pull/8)
|
|
21
|
+
|
|
22
|
+
## 0.1.5 - 2015-03-26
|
|
23
|
+
|
|
24
|
+
* [new feature] Added cluster_name parameter [#7](https://github.com/muga/embulk-output-elasticsearch/pull/7)
|
|
25
|
+
|
|
26
|
+
## 0.1.4 - 2015-03-19
|
|
27
|
+
|
|
28
|
+
* [maintenance] Fixed parameter names index_name to index, doc_id_column to id. [#5](https://github.com/muga/embulk-output-elasticsearch/pull/5)
|
|
29
|
+
* [maintenance] Fixed typo at parameter [#6](https://github.com/muga/embulk-output-elasticsearch/pull/6)
|
|
30
|
+
|
|
31
|
+
## 0.1.3 - 2015-02-25
|
|
32
|
+
|
|
33
|
+
* [new feature] Supported timestamp column [#4](https://github.com/muga/embulk-output-elasticsearch/pull/4)
|
|
34
|
+
|
|
35
|
+
## 0.1.2 - 2015-02-24
|
|
36
|
+
|
|
37
|
+
## 0.1.1 - 2015-02-16
|
|
38
|
+
|
|
39
|
+
## 0.1.0 - 2015-02-16
|
data/README.md
CHANGED
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
## Configuration
|
|
11
11
|
|
|
12
|
+
- **mode**: "insert" or "replace". See below(string, optional, default is insert)
|
|
12
13
|
- **nodes**: list of nodes. nodes are pairs of host and port (list, required)
|
|
13
14
|
- **cluster_name**: name of the cluster (string, default is "elasticsearch")
|
|
14
15
|
- **index**: index name (string, required)
|
|
@@ -18,11 +19,38 @@
|
|
|
18
19
|
- **bulk_size**: Sets when to flush a new bulk request based on the size of actions currently added. (long, default is 5242880)
|
|
19
20
|
- **concurrent_requests**: concurrent_requests (int, default is 5)
|
|
20
21
|
|
|
22
|
+
### Modes
|
|
23
|
+
|
|
24
|
+
#### insert:
|
|
25
|
+
|
|
26
|
+
default.
|
|
27
|
+
This mode writes data to existing index.
|
|
28
|
+
|
|
29
|
+
#### replace:
|
|
30
|
+
|
|
31
|
+
1. Create new temporary index
|
|
32
|
+
2. Insert data into the new index
|
|
33
|
+
3. replace the alias with the new index. If alias doesn't exists, plugin will create new alias.
|
|
34
|
+
4. Delete existing (old) index if exists
|
|
35
|
+
|
|
36
|
+
Index should not exists with the same name as the alias
|
|
37
|
+
|
|
38
|
+
```yaml
|
|
39
|
+
out:
|
|
40
|
+
type: elasticsearch
|
|
41
|
+
mode: replace
|
|
42
|
+
nodes:
|
|
43
|
+
- {host: localhost, port: 9300}
|
|
44
|
+
index: <alias name> # plugin generates index name like <index>_%Y%m%d-%H%M%S
|
|
45
|
+
index_type: <index type>
|
|
46
|
+
```
|
|
47
|
+
|
|
21
48
|
## Example
|
|
22
49
|
|
|
23
50
|
```yaml
|
|
24
51
|
out:
|
|
25
52
|
type: elasticsearch
|
|
53
|
+
mode: insert
|
|
26
54
|
nodes:
|
|
27
55
|
- {host: localhost, port: 9300}
|
|
28
56
|
index: <index name>
|
|
@@ -32,5 +60,53 @@ out:
|
|
|
32
60
|
## Build
|
|
33
61
|
|
|
34
62
|
```
|
|
35
|
-
$ ./gradlew gem
|
|
63
|
+
$ ./gradlew gem # -t to watch change of files and rebuild continuously
|
|
36
64
|
```
|
|
65
|
+
|
|
66
|
+
## Test
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
$ ./gradlew test # -t to watch change of files and rebuild continuously
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
To run unit tests, we need to configure the following environment variables.
|
|
73
|
+
|
|
74
|
+
When environment variables are not set, skip almost test cases.
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
ES_HOST
|
|
78
|
+
ES_PORT(optional, if needed, default: 9300)
|
|
79
|
+
ES_INDEX
|
|
80
|
+
ES_INDEX_TYPE
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
If you're using Mac OS X El Capitan and GUI Applications(IDE), like as follows.
|
|
84
|
+
```
|
|
85
|
+
$ vi ~/Library/LaunchAgents/environment.plist
|
|
86
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
87
|
+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
|
88
|
+
<plist version="1.0">
|
|
89
|
+
<dict>
|
|
90
|
+
<key>Label</key>
|
|
91
|
+
<string>my.startup</string>
|
|
92
|
+
<key>ProgramArguments</key>
|
|
93
|
+
<array>
|
|
94
|
+
<string>sh</string>
|
|
95
|
+
<string>-c</string>
|
|
96
|
+
<string>
|
|
97
|
+
launchctl setenv ES_HOST example.com
|
|
98
|
+
launchctl setenv ES_PORT 9300
|
|
99
|
+
launchctl setenv ES_INDEX embulk
|
|
100
|
+
launchctl setenv ES_INDEX_TYPE embulk
|
|
101
|
+
</string>
|
|
102
|
+
</array>
|
|
103
|
+
<key>RunAtLoad</key>
|
|
104
|
+
<true/>
|
|
105
|
+
</dict>
|
|
106
|
+
</plist>
|
|
107
|
+
|
|
108
|
+
$ launchctl load ~/Library/LaunchAgents/environment.plist
|
|
109
|
+
$ launchctl getenv ES_INDEX //try to get value.
|
|
110
|
+
|
|
111
|
+
Then start your applications.
|
|
112
|
+
```
|
data/build.gradle
CHANGED
|
@@ -2,6 +2,7 @@ plugins {
|
|
|
2
2
|
id "com.jfrog.bintray" version "1.1"
|
|
3
3
|
id "com.github.jruby-gradle.base" version "0.1.5"
|
|
4
4
|
id "java"
|
|
5
|
+
id "jacoco"
|
|
5
6
|
}
|
|
6
7
|
import com.github.jrubygradle.JRubyExec
|
|
7
8
|
repositories {
|
|
@@ -13,18 +14,20 @@ configurations {
|
|
|
13
14
|
provided
|
|
14
15
|
}
|
|
15
16
|
|
|
16
|
-
version = "0.
|
|
17
|
+
version = "0.2.0"
|
|
17
18
|
|
|
18
19
|
compileJava.options.encoding = 'UTF-8' // source encoding
|
|
19
20
|
sourceCompatibility = 1.7
|
|
20
21
|
targetCompatibility = 1.7
|
|
21
22
|
|
|
22
23
|
dependencies {
|
|
23
|
-
compile "org.embulk:embulk-core:0.7.
|
|
24
|
-
provided "org.embulk:embulk-core:0.7.
|
|
25
|
-
compile 'org.elasticsearch:elasticsearch:
|
|
24
|
+
compile "org.embulk:embulk-core:0.7.7"
|
|
25
|
+
provided "org.embulk:embulk-core:0.7.7"
|
|
26
|
+
compile 'org.elasticsearch:elasticsearch:2.0.0'
|
|
27
|
+
|
|
26
28
|
testCompile "junit:junit:4.+"
|
|
27
|
-
testCompile "org.
|
|
29
|
+
testCompile "org.embulk:embulk-core:0.7.7:tests"
|
|
30
|
+
testCompile "org.embulk:embulk-standards:0.7.7"
|
|
28
31
|
}
|
|
29
32
|
|
|
30
33
|
task classpath(type: Copy, dependsOn: ["jar"]) {
|
|
@@ -67,4 +70,12 @@ Gem::Specification.new do |spec|
|
|
|
67
70
|
spec.add_development_dependency "test-unit", ["~> 3.0.2"]
|
|
68
71
|
end
|
|
69
72
|
/$)
|
|
70
|
-
}
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
jacocoTestReport {
|
|
76
|
+
afterEvaluate {
|
|
77
|
+
classDirectories = files(classDirectories.files.collect {
|
|
78
|
+
fileTree(dir: it, exclude: 'org/embulk/output/elasticsearch/ElasticsearchOutputPlugin$1.class')
|
|
79
|
+
})
|
|
80
|
+
}
|
|
81
|
+
}
|
|
@@ -1,9 +1,18 @@
|
|
|
1
1
|
package org.embulk.output.elasticsearch;
|
|
2
2
|
|
|
3
|
-
import
|
|
4
|
-
import
|
|
5
|
-
import
|
|
6
|
-
import
|
|
3
|
+
import java.io.IOException;
|
|
4
|
+
import java.text.SimpleDateFormat;
|
|
5
|
+
import java.util.ArrayList;
|
|
6
|
+
import java.util.Date;
|
|
7
|
+
import java.util.List;
|
|
8
|
+
import java.util.Locale;
|
|
9
|
+
import java.util.concurrent.TimeUnit;
|
|
10
|
+
import java.net.InetAddress;
|
|
11
|
+
import java.net.UnknownHostException;
|
|
12
|
+
|
|
13
|
+
import com.fasterxml.jackson.annotation.JsonCreator;
|
|
14
|
+
import com.fasterxml.jackson.annotation.JsonValue;
|
|
15
|
+
|
|
7
16
|
import org.elasticsearch.action.bulk.BulkItemResponse;
|
|
8
17
|
import org.elasticsearch.action.bulk.BulkProcessor;
|
|
9
18
|
import org.elasticsearch.action.bulk.BulkRequest;
|
|
@@ -12,38 +21,44 @@ import org.elasticsearch.action.index.IndexRequest;
|
|
|
12
21
|
import org.elasticsearch.client.Client;
|
|
13
22
|
import org.elasticsearch.client.Requests;
|
|
14
23
|
import org.elasticsearch.client.transport.TransportClient;
|
|
15
|
-
import org.elasticsearch.
|
|
16
|
-
import org.elasticsearch.
|
|
17
|
-
import org.elasticsearch.common.
|
|
24
|
+
import org.elasticsearch.cluster.metadata.AliasMetaData;
|
|
25
|
+
import org.elasticsearch.cluster.metadata.AliasOrIndex;
|
|
26
|
+
import org.elasticsearch.common.collect.ImmutableOpenMap;
|
|
18
27
|
import org.elasticsearch.common.settings.Settings;
|
|
19
28
|
import org.elasticsearch.common.transport.InetSocketTransportAddress;
|
|
29
|
+
import org.elasticsearch.common.unit.ByteSizeValue;
|
|
20
30
|
import org.elasticsearch.common.xcontent.XContentBuilder;
|
|
21
31
|
import org.elasticsearch.common.xcontent.XContentFactory;
|
|
22
|
-
import org.elasticsearch.
|
|
23
|
-
import org.elasticsearch.
|
|
24
|
-
import org.
|
|
32
|
+
import org.elasticsearch.action.admin.cluster.state.ClusterStateRequest;
|
|
33
|
+
import org.elasticsearch.action.admin.indices.alias.get.GetAliasesRequest;
|
|
34
|
+
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
|
|
35
|
+
import org.elasticsearch.index.IndexNotFoundException;
|
|
36
|
+
import org.elasticsearch.indices.InvalidAliasNameException;
|
|
37
|
+
|
|
25
38
|
import org.embulk.config.Config;
|
|
26
39
|
import org.embulk.config.ConfigDefault;
|
|
27
40
|
import org.embulk.config.ConfigDiff;
|
|
41
|
+
import org.embulk.config.ConfigException;
|
|
28
42
|
import org.embulk.config.ConfigSource;
|
|
29
43
|
import org.embulk.config.Task;
|
|
44
|
+
import org.embulk.config.TaskReport;
|
|
30
45
|
import org.embulk.config.TaskSource;
|
|
31
46
|
import org.embulk.spi.Column;
|
|
47
|
+
import org.embulk.spi.ColumnVisitor;
|
|
32
48
|
import org.embulk.spi.Exec;
|
|
33
49
|
import org.embulk.spi.OutputPlugin;
|
|
34
50
|
import org.embulk.spi.Page;
|
|
35
51
|
import org.embulk.spi.PageReader;
|
|
36
52
|
import org.embulk.spi.Schema;
|
|
37
|
-
import org.embulk.spi.ColumnVisitor;
|
|
38
53
|
import org.embulk.spi.TransactionalPageOutput;
|
|
54
|
+
import org.embulk.spi.time.Timestamp;
|
|
55
|
+
import org.embulk.spi.type.Types;
|
|
39
56
|
import org.slf4j.Logger;
|
|
40
57
|
|
|
41
|
-
import
|
|
42
|
-
import
|
|
43
|
-
import
|
|
44
|
-
import
|
|
45
|
-
|
|
46
|
-
import static com.google.common.base.Preconditions.checkState;
|
|
58
|
+
import com.google.common.base.Optional;
|
|
59
|
+
import com.google.common.base.Throwables;
|
|
60
|
+
import com.google.inject.Inject;
|
|
61
|
+
import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
|
|
47
62
|
|
|
48
63
|
public class ElasticsearchOutputPlugin
|
|
49
64
|
implements OutputPlugin
|
|
@@ -62,6 +77,10 @@ public class ElasticsearchOutputPlugin
|
|
|
62
77
|
public interface PluginTask
|
|
63
78
|
extends Task
|
|
64
79
|
{
|
|
80
|
+
@Config("mode")
|
|
81
|
+
@ConfigDefault("\"insert\"")
|
|
82
|
+
public Mode getMode();
|
|
83
|
+
|
|
65
84
|
@Config("nodes")
|
|
66
85
|
public List<NodeAddressTask> getNodes();
|
|
67
86
|
|
|
@@ -71,6 +90,12 @@ public class ElasticsearchOutputPlugin
|
|
|
71
90
|
|
|
72
91
|
@Config("index")
|
|
73
92
|
public String getIndex();
|
|
93
|
+
public void setIndex(String indexName);
|
|
94
|
+
|
|
95
|
+
@Config("alias")
|
|
96
|
+
@ConfigDefault("null")
|
|
97
|
+
public Optional<String> getAlias();
|
|
98
|
+
public void setAlias(Optional<String> aliasName);
|
|
74
99
|
|
|
75
100
|
@Config("index_type")
|
|
76
101
|
public String getType();
|
|
@@ -108,21 +133,15 @@ public class ElasticsearchOutputPlugin
|
|
|
108
133
|
|
|
109
134
|
// confirm that a client can be initialized
|
|
110
135
|
try (Client client = createClient(task)) {
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
for (Column column : schema.getColumns()) {
|
|
118
|
-
if (column.equals(id)) {
|
|
119
|
-
found = true;
|
|
136
|
+
log.info(String.format("Executing plugin with '%s' mode.", task.getMode()));
|
|
137
|
+
if (task.getMode().equals(Mode.REPLACE)) {
|
|
138
|
+
task.setAlias(Optional.of(task.getIndex()));
|
|
139
|
+
task.setIndex(generateNewIndexName(task.getIndex()));
|
|
140
|
+
if (isExistsIndex(task.getAlias().orNull(), client) && !isAlias(task.getAlias().orNull(), client)) {
|
|
141
|
+
throw new ConfigException(String.format("Invalid alias name [%s], an index exists with the same name as the alias", task.getAlias().orNull()));
|
|
120
142
|
}
|
|
121
143
|
}
|
|
122
|
-
|
|
123
|
-
}
|
|
124
|
-
|
|
125
|
-
try {
|
|
144
|
+
log.info(String.format("Inserting data into index[%s]", task.getIndex()));
|
|
126
145
|
control.run(task.dump());
|
|
127
146
|
} catch (Exception e) {
|
|
128
147
|
throw Throwables.propagate(e);
|
|
@@ -145,19 +164,31 @@ public class ElasticsearchOutputPlugin
|
|
|
145
164
|
public void cleanup(TaskSource taskSource,
|
|
146
165
|
Schema schema, int processorCount,
|
|
147
166
|
List<TaskReport> successTaskReports)
|
|
148
|
-
{
|
|
167
|
+
{
|
|
168
|
+
final PluginTask task = taskSource.loadTask(PluginTask.class);
|
|
169
|
+
if (task.getMode().equals(Mode.REPLACE)) {
|
|
170
|
+
try (Client client = createClient(task)) {
|
|
171
|
+
reAssignAlias(task.getAlias().orNull(), task.getIndex(), client);
|
|
172
|
+
} catch (IndexNotFoundException | InvalidAliasNameException e) {
|
|
173
|
+
throw new ConfigException(e);
|
|
174
|
+
}
|
|
175
|
+
}
|
|
176
|
+
}
|
|
149
177
|
|
|
150
178
|
private Client createClient(final PluginTask task)
|
|
151
179
|
{
|
|
152
180
|
// @see http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/client.html
|
|
153
|
-
Settings settings =
|
|
154
|
-
.classLoader(Settings.class.getClassLoader())
|
|
181
|
+
Settings settings = Settings.settingsBuilder()
|
|
155
182
|
.put("cluster.name", task.getClusterName())
|
|
156
183
|
.build();
|
|
157
|
-
TransportClient client =
|
|
184
|
+
TransportClient client = TransportClient.builder().settings(settings).build();
|
|
158
185
|
List<NodeAddressTask> nodes = task.getNodes();
|
|
159
186
|
for (NodeAddressTask node : nodes) {
|
|
160
|
-
|
|
187
|
+
try {
|
|
188
|
+
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(node.getHost()), node.getPort()));
|
|
189
|
+
} catch (UnknownHostException e) {
|
|
190
|
+
Throwables.propagate(e);
|
|
191
|
+
}
|
|
161
192
|
}
|
|
162
193
|
return client;
|
|
163
194
|
}
|
|
@@ -208,7 +239,6 @@ public class ElasticsearchOutputPlugin
|
|
|
208
239
|
int processorIndex)
|
|
209
240
|
{
|
|
210
241
|
final PluginTask task = taskSource.loadTask(PluginTask.class);
|
|
211
|
-
|
|
212
242
|
Client client = createClient(task);
|
|
213
243
|
BulkProcessor bulkProcessor = newBulkProcessor(task, client);
|
|
214
244
|
ElasticsearchPageOutput pageOutput = new ElasticsearchPageOutput(task, client, bulkProcessor);
|
|
@@ -224,6 +254,7 @@ public class ElasticsearchOutputPlugin
|
|
|
224
254
|
private BulkProcessor bulkProcessor;
|
|
225
255
|
|
|
226
256
|
private PageReader pageReader;
|
|
257
|
+
private Column idColumn;
|
|
227
258
|
|
|
228
259
|
private final String index;
|
|
229
260
|
private final String type;
|
|
@@ -244,6 +275,7 @@ public class ElasticsearchOutputPlugin
|
|
|
244
275
|
void open(final Schema schema)
|
|
245
276
|
{
|
|
246
277
|
pageReader = new PageReader(schema);
|
|
278
|
+
idColumn = (id == null) ? null : schema.lookupColumn(id);
|
|
247
279
|
}
|
|
248
280
|
|
|
249
281
|
@Override
|
|
@@ -342,7 +374,7 @@ public class ElasticsearchOutputPlugin
|
|
|
342
374
|
});
|
|
343
375
|
|
|
344
376
|
contextBuilder.endObject();
|
|
345
|
-
bulkProcessor.add(newIndexRequest().source(contextBuilder));
|
|
377
|
+
bulkProcessor.add(newIndexRequest(getIdValue(idColumn)).source(contextBuilder));
|
|
346
378
|
|
|
347
379
|
} catch (IOException e) {
|
|
348
380
|
Throwables.propagate(e); // TODO error handling
|
|
@@ -350,9 +382,33 @@ public class ElasticsearchOutputPlugin
|
|
|
350
382
|
}
|
|
351
383
|
}
|
|
352
384
|
|
|
353
|
-
|
|
385
|
+
/**
|
|
386
|
+
* @param inputColumn
|
|
387
|
+
* @return
|
|
388
|
+
*/
|
|
389
|
+
private String getIdValue(Column inputColumn) {
|
|
390
|
+
if (inputColumn == null) return null;
|
|
391
|
+
if (pageReader.isNull(inputColumn)) return null;
|
|
392
|
+
String idValue = null;
|
|
393
|
+
if (Types.STRING.equals(inputColumn.getType())) {
|
|
394
|
+
idValue = pageReader.getString(inputColumn);
|
|
395
|
+
} else if (Types.BOOLEAN.equals(inputColumn.getType())) {
|
|
396
|
+
idValue = pageReader.getBoolean(inputColumn) + "";
|
|
397
|
+
} else if (Types.DOUBLE.equals(inputColumn.getType())) {
|
|
398
|
+
idValue = pageReader.getDouble(inputColumn) + "";
|
|
399
|
+
} else if (Types.LONG.equals(inputColumn.getType())) {
|
|
400
|
+
idValue = pageReader.getLong(inputColumn) + "";
|
|
401
|
+
} else if (Types.TIMESTAMP.equals(inputColumn.getType())) {
|
|
402
|
+
idValue = pageReader.getTimestamp(inputColumn).toString();
|
|
403
|
+
} else {
|
|
404
|
+
idValue = null;
|
|
405
|
+
}
|
|
406
|
+
return idValue;
|
|
407
|
+
}
|
|
408
|
+
|
|
409
|
+
private IndexRequest newIndexRequest(String idValue)
|
|
354
410
|
{
|
|
355
|
-
return Requests.indexRequest(index).type(type).id(
|
|
411
|
+
return Requests.indexRequest(index).type(type).id(idValue);
|
|
356
412
|
}
|
|
357
413
|
|
|
358
414
|
@Override
|
|
@@ -400,4 +456,91 @@ public class ElasticsearchOutputPlugin
|
|
|
400
456
|
}
|
|
401
457
|
|
|
402
458
|
}
|
|
459
|
+
|
|
460
|
+
public enum Mode
|
|
461
|
+
{
|
|
462
|
+
INSERT,
|
|
463
|
+
REPLACE;
|
|
464
|
+
|
|
465
|
+
@JsonValue
|
|
466
|
+
@Override
|
|
467
|
+
public String toString()
|
|
468
|
+
{
|
|
469
|
+
return name().toLowerCase(Locale.ENGLISH);
|
|
470
|
+
}
|
|
471
|
+
|
|
472
|
+
@JsonCreator
|
|
473
|
+
public static Mode fromString(String value)
|
|
474
|
+
{
|
|
475
|
+
switch (value) {
|
|
476
|
+
case "insert":
|
|
477
|
+
return INSERT;
|
|
478
|
+
case "replace":
|
|
479
|
+
return REPLACE;
|
|
480
|
+
default:
|
|
481
|
+
throw new ConfigException(String.format("Unknown mode '%s'. Supported modes are insert, truncate_insert, replace", value));
|
|
482
|
+
}
|
|
483
|
+
}
|
|
484
|
+
}
|
|
485
|
+
|
|
486
|
+
private void reAssignAlias(String aliasName, String newIndexName, Client client)
|
|
487
|
+
throws IndexNotFoundException, InvalidAliasNameException
|
|
488
|
+
{
|
|
489
|
+
if (!isExistsAlias(aliasName, client)) {
|
|
490
|
+
client.admin().indices().prepareAliases()
|
|
491
|
+
.addAlias(newIndexName, aliasName)
|
|
492
|
+
.execute().actionGet();
|
|
493
|
+
log.info(String.format("Assigned alias[%s] to index[%s]", aliasName, newIndexName));
|
|
494
|
+
} else {
|
|
495
|
+
List<String> oldIndices = getIndexByAlias(aliasName, client);
|
|
496
|
+
client.admin().indices().prepareAliases()
|
|
497
|
+
.removeAlias(oldIndices.toArray(new String[oldIndices.size()]), aliasName)
|
|
498
|
+
.addAlias(newIndexName, aliasName)
|
|
499
|
+
.execute().actionGet();
|
|
500
|
+
log.info(String.format("Reassigned alias[%s] from index%s to index[%s]", aliasName, oldIndices, newIndexName));
|
|
501
|
+
for (String index : oldIndices) {
|
|
502
|
+
deleteIndex(index, client);
|
|
503
|
+
}
|
|
504
|
+
}
|
|
505
|
+
}
|
|
506
|
+
|
|
507
|
+
private void deleteIndex(String indexName, Client client)
|
|
508
|
+
{
|
|
509
|
+
client.admin().indices().delete(new DeleteIndexRequest(indexName)).actionGet();
|
|
510
|
+
log.info(String.format("Deleted Index [%s]", indexName));
|
|
511
|
+
}
|
|
512
|
+
|
|
513
|
+
private List<String> getIndexByAlias(String aliasName, Client client)
|
|
514
|
+
{
|
|
515
|
+
ImmutableOpenMap<String, List<AliasMetaData>> map = client.admin().indices().getAliases(new GetAliasesRequest(aliasName))
|
|
516
|
+
.actionGet().getAliases();
|
|
517
|
+
List<String> indices = new ArrayList<>();
|
|
518
|
+
for (ObjectObjectCursor<String, List<AliasMetaData>> c : map) {
|
|
519
|
+
indices.add(c.key);
|
|
520
|
+
}
|
|
521
|
+
|
|
522
|
+
return indices;
|
|
523
|
+
}
|
|
524
|
+
|
|
525
|
+
private boolean isExistsAlias(String aliasName, Client client)
|
|
526
|
+
{
|
|
527
|
+
return client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().hasAlias(aliasName);
|
|
528
|
+
}
|
|
529
|
+
|
|
530
|
+
private boolean isExistsIndex(String indexName, Client client)
|
|
531
|
+
{
|
|
532
|
+
return client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().hasIndex(indexName);
|
|
533
|
+
}
|
|
534
|
+
|
|
535
|
+
private boolean isAlias(String aliasName, Client client)
|
|
536
|
+
{
|
|
537
|
+
AliasOrIndex aliasOrIndex = client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().getAliasAndIndexLookup().get(aliasName);
|
|
538
|
+
return aliasOrIndex != null && aliasOrIndex.isAlias();
|
|
539
|
+
}
|
|
540
|
+
|
|
541
|
+
public String generateNewIndexName(String indexName)
|
|
542
|
+
{
|
|
543
|
+
Timestamp time = Exec.getTransactionTime();
|
|
544
|
+
return indexName + new SimpleDateFormat("_yyyyMMdd-HHmmss").format(time.toEpochMilli());
|
|
545
|
+
}
|
|
403
546
|
}
|
|
@@ -1,5 +1,484 @@
|
|
|
1
1
|
package org.embulk.output.elasticsearch;
|
|
2
2
|
|
|
3
|
+
import java.io.ByteArrayOutputStream;
|
|
4
|
+
import java.io.InputStream;
|
|
5
|
+
import java.io.IOException;
|
|
6
|
+
import java.lang.reflect.InvocationTargetException;
|
|
7
|
+
import java.lang.reflect.Method;
|
|
8
|
+
import java.net.UnknownHostException;
|
|
9
|
+
import java.text.ParseException;
|
|
10
|
+
import java.text.SimpleDateFormat;
|
|
11
|
+
import java.util.Arrays;
|
|
12
|
+
import java.util.List;
|
|
13
|
+
import java.util.Map;
|
|
14
|
+
import java.security.GeneralSecurityException;
|
|
15
|
+
|
|
16
|
+
import com.google.common.collect.Lists;
|
|
17
|
+
import com.google.common.collect.ImmutableList;
|
|
18
|
+
import com.google.common.collect.ImmutableMap;
|
|
19
|
+
import org.elasticsearch.action.admin.cluster.state.ClusterStateRequest;
|
|
20
|
+
import org.elasticsearch.action.admin.indices.create.CreateIndexRequest;
|
|
21
|
+
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
|
|
22
|
+
import org.elasticsearch.action.get.GetResponse;
|
|
23
|
+
import org.elasticsearch.client.Client;
|
|
24
|
+
import org.embulk.EmbulkTestRuntime;
|
|
25
|
+
import org.embulk.config.ConfigException;
|
|
26
|
+
import org.embulk.config.TaskReport;
|
|
27
|
+
import org.embulk.config.TaskSource;
|
|
28
|
+
import org.embulk.config.ConfigSource;
|
|
29
|
+
import org.embulk.spi.Exec;
|
|
30
|
+
import org.embulk.spi.OutputPlugin;
|
|
31
|
+
import org.embulk.spi.Page;
|
|
32
|
+
import org.embulk.spi.PageTestUtils;
|
|
33
|
+
import org.embulk.spi.Schema;
|
|
34
|
+
import org.embulk.spi.time.Timestamp;
|
|
35
|
+
import org.embulk.spi.TransactionalPageOutput;
|
|
36
|
+
import org.embulk.spi.TestPageBuilderReader.MockPageOutput;
|
|
37
|
+
import org.embulk.standards.CsvParserPlugin;
|
|
38
|
+
import org.embulk.output.elasticsearch.ElasticsearchOutputPlugin.PluginTask;
|
|
39
|
+
import org.junit.Before;
|
|
40
|
+
import org.junit.BeforeClass;
|
|
41
|
+
import org.junit.Rule;
|
|
42
|
+
import org.junit.Test;
|
|
43
|
+
import static org.junit.Assert.assertEquals;
|
|
44
|
+
import static org.junit.Assert.assertTrue;
|
|
45
|
+
import static org.junit.Assume.assumeNotNull;
|
|
46
|
+
|
|
3
47
|
public class TestElasticsearchOutputPlugin
|
|
4
48
|
{
|
|
49
|
+
private static String ES_HOST;
|
|
50
|
+
private static int ES_PORT;
|
|
51
|
+
private static List ES_NODES;
|
|
52
|
+
private static String ES_CLUSTER_NAME;
|
|
53
|
+
private static String ES_INDEX;
|
|
54
|
+
private static String ES_INDEX_TYPE;
|
|
55
|
+
private static String ES_ID;
|
|
56
|
+
private static int ES_BULK_ACTIONS;
|
|
57
|
+
private static int ES_BULK_SIZE;
|
|
58
|
+
private static int ES_CONCURRENT_REQUESTS;
|
|
59
|
+
private static String PATH_PREFIX;
|
|
60
|
+
|
|
61
|
+
private MockPageOutput pageOutput;
|
|
62
|
+
|
|
63
|
+
final String ES_TEST_INDEX = "index_for_unittest";
|
|
64
|
+
final String ES_TEST_INDEX2 = "index_for_unittest2";
|
|
65
|
+
final String ES_TEST_ALIAS = "alias_for_unittest";
|
|
66
|
+
|
|
67
|
+
/*
|
|
68
|
+
* This test case requires environment variables
|
|
69
|
+
* ES_HOST
|
|
70
|
+
* ES_INDEX
|
|
71
|
+
* ES_INDEX_TYPE
|
|
72
|
+
*/
|
|
73
|
+
@BeforeClass
|
|
74
|
+
public static void initializeConstant()
|
|
75
|
+
{
|
|
76
|
+
ES_HOST = System.getenv("ES_HOST") != null ? System.getenv("ES_HOST") : "";
|
|
77
|
+
ES_PORT = System.getenv("ES_PORT") != null ? Integer.valueOf(System.getenv("ES_PORT")) : 9300;
|
|
78
|
+
|
|
79
|
+
ES_CLUSTER_NAME = System.getenv("ES_CLUSTER_NAME") != null ? System.getenv("ES_CLUSTER_NAME") : "";
|
|
80
|
+
ES_INDEX = System.getenv("ES_INDEX");
|
|
81
|
+
ES_INDEX_TYPE = System.getenv("ES_INDEX_TYPE");
|
|
82
|
+
ES_ID = "id";
|
|
83
|
+
ES_BULK_ACTIONS = System.getenv("ES_BULK_ACTIONS") != null ? Integer.valueOf(System.getenv("ES_BULK_ACTIONS")) : 1000;
|
|
84
|
+
ES_BULK_SIZE = System.getenv("ES_BULK_SIZE") != null ? Integer.valueOf(System.getenv("ES_BULK_SIZE")) : 5242880;
|
|
85
|
+
ES_CONCURRENT_REQUESTS = System.getenv("ES_CONCURRENT_REQUESTS") != null ? Integer.valueOf(System.getenv("ES_CONCURRENT_REQUESTS")) : 5;
|
|
86
|
+
|
|
87
|
+
assumeNotNull(ES_HOST, ES_INDEX, ES_INDEX_TYPE);
|
|
88
|
+
|
|
89
|
+
ES_NODES = Arrays.asList(ImmutableMap.of("host", ES_HOST, "port", ES_PORT));
|
|
90
|
+
|
|
91
|
+
PATH_PREFIX = ElasticsearchOutputPlugin.class.getClassLoader().getResource("sample_01.csv").getPath();
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
@Rule
|
|
96
|
+
public EmbulkTestRuntime runtime = new EmbulkTestRuntime();
|
|
97
|
+
private ElasticsearchOutputPlugin plugin;
|
|
98
|
+
|
|
99
|
+
@Before
|
|
100
|
+
public void createResources()
|
|
101
|
+
throws GeneralSecurityException, NoSuchMethodException,
|
|
102
|
+
IllegalAccessException, InvocationTargetException
|
|
103
|
+
{
|
|
104
|
+
ConfigSource config = config();
|
|
105
|
+
plugin = new ElasticsearchOutputPlugin();
|
|
106
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
107
|
+
pageOutput = new MockPageOutput();
|
|
108
|
+
|
|
109
|
+
Method createClient = ElasticsearchOutputPlugin.class.getDeclaredMethod("createClient", PluginTask.class);
|
|
110
|
+
createClient.setAccessible(true);
|
|
111
|
+
try (Client client = (Client) createClient.invoke(plugin, task)) {
|
|
112
|
+
// Delete alias
|
|
113
|
+
if (client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().hasAlias(ES_TEST_ALIAS)) {
|
|
114
|
+
client.admin().indices().delete(new DeleteIndexRequest(ES_TEST_ALIAS)).actionGet();
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
// Delete index
|
|
118
|
+
if (client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().hasIndex(ES_TEST_INDEX)) {
|
|
119
|
+
client.admin().indices().delete(new DeleteIndexRequest(ES_TEST_INDEX)).actionGet();
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
if (client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().hasIndex(ES_TEST_INDEX2)) {
|
|
123
|
+
client.admin().indices().delete(new DeleteIndexRequest(ES_TEST_INDEX2)).actionGet();
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
@Test
|
|
129
|
+
public void testDefaultValues()
|
|
130
|
+
{
|
|
131
|
+
ConfigSource config = config();
|
|
132
|
+
ElasticsearchOutputPlugin.PluginTask task = config.loadConfig(PluginTask.class);
|
|
133
|
+
assertEquals(ES_INDEX, task.getIndex());
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
@Test
|
|
137
|
+
public void testDefaultValuesNull()
|
|
138
|
+
{
|
|
139
|
+
ConfigSource config = Exec.newConfigSource()
|
|
140
|
+
.set("in", inputConfig())
|
|
141
|
+
.set("parser", parserConfig(schemaConfig()))
|
|
142
|
+
.set("type", "elasticsearch")
|
|
143
|
+
.set("mode", "") // NULL
|
|
144
|
+
.set("nodes", ES_NODES)
|
|
145
|
+
.set("cluster_name", ES_CLUSTER_NAME)
|
|
146
|
+
.set("index", ES_INDEX)
|
|
147
|
+
.set("index_type", ES_INDEX_TYPE)
|
|
148
|
+
.set("id", ES_ID)
|
|
149
|
+
.set("bulk_actions", ES_BULK_ACTIONS)
|
|
150
|
+
.set("bulk_size", ES_BULK_SIZE)
|
|
151
|
+
.set("concurrent_requests", ES_CONCURRENT_REQUESTS
|
|
152
|
+
);
|
|
153
|
+
Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();
|
|
154
|
+
try {
|
|
155
|
+
plugin.transaction(config, schema, 0, new OutputPlugin.Control()
|
|
156
|
+
{
|
|
157
|
+
@Override
|
|
158
|
+
public List<TaskReport> run(TaskSource taskSource)
|
|
159
|
+
{
|
|
160
|
+
return Lists.newArrayList(Exec.newTaskReport());
|
|
161
|
+
}
|
|
162
|
+
});
|
|
163
|
+
} catch (Throwable t) {
|
|
164
|
+
if (t instanceof RuntimeException) {
|
|
165
|
+
assertTrue(t.getCause().getCause() instanceof ConfigException);
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
@Test
|
|
171
|
+
public void testResume()
|
|
172
|
+
{
|
|
173
|
+
ConfigSource config = config();
|
|
174
|
+
Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();
|
|
175
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
176
|
+
plugin.resume(task.dump(), schema, 0, new OutputPlugin.Control()
|
|
177
|
+
{
|
|
178
|
+
@Override
|
|
179
|
+
public List<TaskReport> run(TaskSource taskSource)
|
|
180
|
+
{
|
|
181
|
+
return Lists.newArrayList(Exec.newTaskReport());
|
|
182
|
+
}
|
|
183
|
+
});
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
@Test
|
|
187
|
+
public void testTransaction()
|
|
188
|
+
{
|
|
189
|
+
ConfigSource config = Exec.newConfigSource()
|
|
190
|
+
.set("in", inputConfig())
|
|
191
|
+
.set("parser", parserConfig(schemaConfig()))
|
|
192
|
+
.set("type", "elasticsearch")
|
|
193
|
+
.set("mode", "replace")
|
|
194
|
+
.set("nodes", ES_NODES)
|
|
195
|
+
.set("cluster_name", ES_CLUSTER_NAME)
|
|
196
|
+
.set("index", ES_INDEX)
|
|
197
|
+
.set("index_type", ES_INDEX_TYPE)
|
|
198
|
+
.set("id", ES_ID)
|
|
199
|
+
.set("bulk_actions", ES_BULK_ACTIONS)
|
|
200
|
+
.set("bulk_size", ES_BULK_SIZE)
|
|
201
|
+
.set("concurrent_requests", ES_CONCURRENT_REQUESTS
|
|
202
|
+
);
|
|
203
|
+
Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();
|
|
204
|
+
plugin.transaction(config, schema, 0, new OutputPlugin.Control()
|
|
205
|
+
{
|
|
206
|
+
@Override
|
|
207
|
+
public List<TaskReport> run(TaskSource taskSource)
|
|
208
|
+
{
|
|
209
|
+
return Lists.newArrayList(Exec.newTaskReport());
|
|
210
|
+
}
|
|
211
|
+
});
|
|
212
|
+
// no error happens
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
@Test
|
|
216
|
+
public void testOutputByOpen()
|
|
217
|
+
throws GeneralSecurityException, IOException, NoSuchMethodException,
|
|
218
|
+
IllegalAccessException, InvocationTargetException, ParseException
|
|
219
|
+
{
|
|
220
|
+
ConfigSource config = config();
|
|
221
|
+
Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();
|
|
222
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
223
|
+
plugin.transaction(config, schema, 0, new OutputPlugin.Control() {
|
|
224
|
+
@Override
|
|
225
|
+
public List<TaskReport> run(TaskSource taskSource) {
|
|
226
|
+
return Lists.newArrayList(Exec.newTaskReport());
|
|
227
|
+
}
|
|
228
|
+
});
|
|
229
|
+
TransactionalPageOutput output = plugin.open(task.dump(), schema, 0);
|
|
230
|
+
|
|
231
|
+
List<Page> pages = PageTestUtils.buildPage(runtime.getBufferAllocator(), schema, 1L, 32864L, Timestamp.ofEpochSecond(1422386629), Timestamp.ofEpochSecond(1422316800), true, 123.45, "embulk");
|
|
232
|
+
assertEquals(1, pages.size());
|
|
233
|
+
for (Page page : pages) {
|
|
234
|
+
output.add(page);
|
|
235
|
+
}
|
|
236
|
+
|
|
237
|
+
output.finish();
|
|
238
|
+
output.commit();
|
|
239
|
+
|
|
240
|
+
Method createClient = ElasticsearchOutputPlugin.class.getDeclaredMethod("createClient", PluginTask.class);
|
|
241
|
+
createClient.setAccessible(true);
|
|
242
|
+
try (Client client = (Client) createClient.invoke(plugin, task)) {
|
|
243
|
+
GetResponse response = client.prepareGet(ES_INDEX, ES_INDEX_TYPE, "1").execute().actionGet();
|
|
244
|
+
assertTrue(response.isExists());
|
|
245
|
+
if (response.isExists()) {
|
|
246
|
+
Map<String, Object> map = response.getSourceAsMap();
|
|
247
|
+
assertEquals(1, map.get("id"));
|
|
248
|
+
assertEquals(32864, map.get("account"));
|
|
249
|
+
assertEquals("2015-01-27T19:23:49.000Z", map.get("time"));
|
|
250
|
+
assertEquals("2015-01-27T00:00:00.000Z", map.get("purchase"));
|
|
251
|
+
assertEquals(true, map.get("flg"));
|
|
252
|
+
assertEquals(123.45, map.get("score"));
|
|
253
|
+
assertEquals("embulk", map.get("comment"));
|
|
254
|
+
}
|
|
255
|
+
}
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
@Test
|
|
259
|
+
public void testOpenAbort()
|
|
260
|
+
{
|
|
261
|
+
ConfigSource config = config();
|
|
262
|
+
Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();
|
|
263
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
264
|
+
TransactionalPageOutput output = plugin.open(task.dump(), schema, 0);
|
|
265
|
+
output.abort();
|
|
266
|
+
// no error happens.
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
@Test
|
|
270
|
+
public void testCreateClientThrowsException()
|
|
271
|
+
throws GeneralSecurityException, IOException, NoSuchMethodException,
|
|
272
|
+
IllegalAccessException, InvocationTargetException
|
|
273
|
+
{
|
|
274
|
+
ConfigSource config = Exec.newConfigSource()
|
|
275
|
+
.set("in", inputConfig())
|
|
276
|
+
.set("parser", parserConfig(schemaConfig()))
|
|
277
|
+
.set("type", "elasticsearch")
|
|
278
|
+
.set("mode", "replace")
|
|
279
|
+
.set("nodes", Arrays.asList(ImmutableMap.of("host", "unknown-host", "port", 9300)))
|
|
280
|
+
.set("cluster_name", ES_CLUSTER_NAME)
|
|
281
|
+
.set("index", ES_INDEX)
|
|
282
|
+
.set("index_type", ES_INDEX_TYPE)
|
|
283
|
+
.set("id", ES_ID)
|
|
284
|
+
.set("bulk_actions", ES_BULK_ACTIONS)
|
|
285
|
+
.set("bulk_size", ES_BULK_SIZE)
|
|
286
|
+
.set("concurrent_requests", ES_CONCURRENT_REQUESTS
|
|
287
|
+
);
|
|
288
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
289
|
+
|
|
290
|
+
Method createClient = ElasticsearchOutputPlugin.class.getDeclaredMethod("createClient", PluginTask.class);
|
|
291
|
+
createClient.setAccessible(true);
|
|
292
|
+
try (Client client = (Client) createClient.invoke(plugin, task)) {
|
|
293
|
+
} catch (Throwable t) {
|
|
294
|
+
if (t instanceof InvocationTargetException) {
|
|
295
|
+
assertTrue(t.getCause().getCause() instanceof UnknownHostException);
|
|
296
|
+
}
|
|
297
|
+
}
|
|
298
|
+
}
|
|
299
|
+
|
|
300
|
+
@Test
|
|
301
|
+
public void testMode()
|
|
302
|
+
{
|
|
303
|
+
assertEquals(2, ElasticsearchOutputPlugin.Mode.values().length);
|
|
304
|
+
assertEquals(ElasticsearchOutputPlugin.Mode.INSERT, ElasticsearchOutputPlugin.Mode.valueOf("INSERT"));
|
|
305
|
+
}
|
|
306
|
+
|
|
307
|
+
@Test(expected = ConfigException.class)
|
|
308
|
+
public void testModeThrowsConfigException()
|
|
309
|
+
{
|
|
310
|
+
ElasticsearchOutputPlugin.Mode.fromString("non-exists-mode");
|
|
311
|
+
}
|
|
312
|
+
|
|
313
|
+
@Test
|
|
314
|
+
public void testDeleteIndex()
|
|
315
|
+
throws GeneralSecurityException, IOException, NoSuchMethodException,
|
|
316
|
+
IllegalAccessException, InvocationTargetException
|
|
317
|
+
{
|
|
318
|
+
ConfigSource config = config();
|
|
319
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
320
|
+
|
|
321
|
+
Method createClient = ElasticsearchOutputPlugin.class.getDeclaredMethod("createClient", PluginTask.class);
|
|
322
|
+
createClient.setAccessible(true);
|
|
323
|
+
try (Client client = (Client) createClient.invoke(plugin, task)) {
|
|
324
|
+
// Create Index
|
|
325
|
+
client.admin().indices().create(new CreateIndexRequest(ES_TEST_INDEX)).actionGet();
|
|
326
|
+
|
|
327
|
+
Method deleteIndex = ElasticsearchOutputPlugin.class.getDeclaredMethod("deleteIndex", String.class, Client.class);
|
|
328
|
+
deleteIndex.setAccessible(true);
|
|
329
|
+
deleteIndex.invoke(plugin, ES_TEST_INDEX, client);
|
|
330
|
+
|
|
331
|
+
assertEquals(false, client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().hasIndex(ES_TEST_INDEX));
|
|
332
|
+
}
|
|
333
|
+
}
|
|
334
|
+
|
|
335
|
+
@Test
|
|
336
|
+
public void testAlias()
|
|
337
|
+
throws GeneralSecurityException, IOException, NoSuchMethodException,
|
|
338
|
+
IllegalAccessException, InvocationTargetException
|
|
339
|
+
{
|
|
340
|
+
ConfigSource config = config();
|
|
341
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
342
|
+
|
|
343
|
+
Method createClient = ElasticsearchOutputPlugin.class.getDeclaredMethod("createClient", PluginTask.class);
|
|
344
|
+
createClient.setAccessible(true);
|
|
345
|
+
try (Client client = (Client) createClient.invoke(plugin, task)) {
|
|
346
|
+
|
|
347
|
+
Method isAlias = ElasticsearchOutputPlugin.class.getDeclaredMethod("isAlias", String.class, Client.class);
|
|
348
|
+
isAlias.setAccessible(true);
|
|
349
|
+
|
|
350
|
+
Method isExistsAlias = ElasticsearchOutputPlugin.class.getDeclaredMethod("isExistsAlias", String.class, Client.class);
|
|
351
|
+
isExistsAlias.setAccessible(true);
|
|
352
|
+
|
|
353
|
+
Method getIndexByAlias = ElasticsearchOutputPlugin.class.getDeclaredMethod("getIndexByAlias", String.class, Client.class);
|
|
354
|
+
getIndexByAlias.setAccessible(true);
|
|
355
|
+
|
|
356
|
+
Method reAssignAlias = ElasticsearchOutputPlugin.class.getDeclaredMethod("reAssignAlias", String.class, String.class, Client.class);
|
|
357
|
+
reAssignAlias.setAccessible(true);
|
|
358
|
+
|
|
359
|
+
assertEquals(false, isAlias.invoke(plugin, ES_TEST_ALIAS, client));
|
|
360
|
+
assertEquals(false, isExistsAlias.invoke(plugin, ES_TEST_ALIAS, client));
|
|
361
|
+
List<String> indicesBefore = (List<String>) getIndexByAlias.invoke(plugin, ES_TEST_ALIAS, client);
|
|
362
|
+
assertEquals(0, indicesBefore.size());
|
|
363
|
+
|
|
364
|
+
// Create Index
|
|
365
|
+
client.admin().indices().create(new CreateIndexRequest(ES_TEST_INDEX)).actionGet();
|
|
366
|
+
client.admin().indices().create(new CreateIndexRequest(ES_TEST_INDEX2)).actionGet();
|
|
367
|
+
// Assign Alias
|
|
368
|
+
reAssignAlias.invoke(plugin, ES_TEST_ALIAS, ES_TEST_INDEX, client);
|
|
369
|
+
|
|
370
|
+
assertEquals(true, isAlias.invoke(plugin, ES_TEST_ALIAS, client));
|
|
371
|
+
assertEquals(true, isExistsAlias.invoke(plugin, ES_TEST_ALIAS, client));
|
|
372
|
+
List<String> indicesAfter = (List<String>) getIndexByAlias.invoke(plugin, ES_TEST_ALIAS, client);
|
|
373
|
+
assertEquals(1, indicesAfter.size());
|
|
374
|
+
|
|
375
|
+
// ReAssginAlias
|
|
376
|
+
reAssignAlias.invoke(plugin, ES_TEST_ALIAS, ES_TEST_INDEX2, client);
|
|
377
|
+
List<String> indicesReassign = (List<String>) getIndexByAlias.invoke(plugin, ES_TEST_ALIAS, client);
|
|
378
|
+
assertEquals(1, indicesReassign.size());
|
|
379
|
+
}
|
|
380
|
+
}
|
|
381
|
+
|
|
382
|
+
@Test
|
|
383
|
+
public void testIsExistsIndex()
|
|
384
|
+
throws GeneralSecurityException, IOException, NoSuchMethodException,
|
|
385
|
+
IllegalAccessException, InvocationTargetException
|
|
386
|
+
{
|
|
387
|
+
ConfigSource config = config();
|
|
388
|
+
PluginTask task = config.loadConfig(PluginTask.class);
|
|
389
|
+
|
|
390
|
+
Method createClient = ElasticsearchOutputPlugin.class.getDeclaredMethod("createClient", PluginTask.class);
|
|
391
|
+
createClient.setAccessible(true);
|
|
392
|
+
try (Client client = (Client) createClient.invoke(plugin, task)) {
|
|
393
|
+
Method isExistsIndex = ElasticsearchOutputPlugin.class.getDeclaredMethod("isExistsIndex", String.class, Client.class);
|
|
394
|
+
isExistsIndex.setAccessible(true);
|
|
395
|
+
|
|
396
|
+
// Delete index
|
|
397
|
+
if (client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState().getMetaData().hasIndex(ES_TEST_INDEX)) {
|
|
398
|
+
client.admin().indices().delete(new DeleteIndexRequest(ES_TEST_INDEX)).actionGet();
|
|
399
|
+
}
|
|
400
|
+
assertEquals(false, isExistsIndex.invoke(plugin, ES_TEST_INDEX, client));
|
|
401
|
+
|
|
402
|
+
// Create Index
|
|
403
|
+
client.admin().indices().create(new CreateIndexRequest(ES_TEST_INDEX)).actionGet();
|
|
404
|
+
assertEquals(true, isExistsIndex.invoke(plugin, ES_TEST_INDEX, client));
|
|
405
|
+
}
|
|
406
|
+
}
|
|
407
|
+
|
|
408
|
+
@Test
|
|
409
|
+
public void testGenerateNewIndex()
|
|
410
|
+
{
|
|
411
|
+
String newIndexName = plugin.generateNewIndexName(ES_INDEX);
|
|
412
|
+
Timestamp time = Exec.getTransactionTime();
|
|
413
|
+
assertEquals(ES_INDEX + new SimpleDateFormat("_yyyyMMdd-HHmmss").format(time.toEpochMilli()), newIndexName);
|
|
414
|
+
}
|
|
415
|
+
|
|
416
|
+
private byte[] convertInputStreamToByte(InputStream is) throws IOException
|
|
417
|
+
{
|
|
418
|
+
ByteArrayOutputStream bo = new ByteArrayOutputStream();
|
|
419
|
+
byte [] buffer = new byte[1024];
|
|
420
|
+
while(true) {
|
|
421
|
+
int len = is.read(buffer);
|
|
422
|
+
if(len < 0) {
|
|
423
|
+
break;
|
|
424
|
+
}
|
|
425
|
+
bo.write(buffer, 0, len);
|
|
426
|
+
}
|
|
427
|
+
return bo.toByteArray();
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
private ConfigSource config()
|
|
431
|
+
{
|
|
432
|
+
return Exec.newConfigSource()
|
|
433
|
+
.set("in", inputConfig())
|
|
434
|
+
.set("parser", parserConfig(schemaConfig()))
|
|
435
|
+
.set("type", "elasticsearch")
|
|
436
|
+
.set("mode", "insert")
|
|
437
|
+
.set("nodes", ES_NODES)
|
|
438
|
+
.set("cluster_name", ES_CLUSTER_NAME)
|
|
439
|
+
.set("index", ES_INDEX)
|
|
440
|
+
.set("index_type", ES_INDEX_TYPE)
|
|
441
|
+
.set("id", ES_ID)
|
|
442
|
+
.set("bulk_actions", ES_BULK_ACTIONS)
|
|
443
|
+
.set("bulk_size", ES_BULK_SIZE)
|
|
444
|
+
.set("concurrent_requests", ES_CONCURRENT_REQUESTS);
|
|
445
|
+
}
|
|
446
|
+
|
|
447
|
+
private ImmutableMap<String, Object> inputConfig()
|
|
448
|
+
{
|
|
449
|
+
ImmutableMap.Builder<String, Object> builder = new ImmutableMap.Builder<>();
|
|
450
|
+
builder.put("type", "file");
|
|
451
|
+
builder.put("path_prefix", PATH_PREFIX);
|
|
452
|
+
builder.put("last_path", "");
|
|
453
|
+
return builder.build();
|
|
454
|
+
}
|
|
455
|
+
|
|
456
|
+
private ImmutableMap<String, Object> parserConfig(ImmutableList<Object> schemaConfig)
|
|
457
|
+
{
|
|
458
|
+
ImmutableMap.Builder<String, Object> builder = new ImmutableMap.Builder<>();
|
|
459
|
+
builder.put("type", "csv");
|
|
460
|
+
builder.put("newline", "CRLF");
|
|
461
|
+
builder.put("delimiter", ",");
|
|
462
|
+
builder.put("quote", "\"");
|
|
463
|
+
builder.put("escape", "\"");
|
|
464
|
+
builder.put("trim_if_not_quoted", false);
|
|
465
|
+
builder.put("skip_header_lines", 1);
|
|
466
|
+
builder.put("allow_extra_columns", false);
|
|
467
|
+
builder.put("allow_optional_columns", false);
|
|
468
|
+
builder.put("columns", schemaConfig);
|
|
469
|
+
return builder.build();
|
|
470
|
+
}
|
|
471
|
+
|
|
472
|
+
private ImmutableList<Object> schemaConfig()
|
|
473
|
+
{
|
|
474
|
+
ImmutableList.Builder<Object> builder = new ImmutableList.Builder<>();
|
|
475
|
+
builder.add(ImmutableMap.of("name", "id", "type", "long"));
|
|
476
|
+
builder.add(ImmutableMap.of("name", "account", "type", "long"));
|
|
477
|
+
builder.add(ImmutableMap.of("name", "time", "type", "timestamp", "format", "%Y-%m-%d %H:%M:%S"));
|
|
478
|
+
builder.add(ImmutableMap.of("name", "purchase", "type", "timestamp", "format", "%Y%m%d"));
|
|
479
|
+
builder.add(ImmutableMap.of("name", "flg", "type", "boolean"));
|
|
480
|
+
builder.add(ImmutableMap.of("name", "score", "type", "double"));
|
|
481
|
+
builder.add(ImmutableMap.of("name", "comment", "type", "string"));
|
|
482
|
+
return builder.build();
|
|
483
|
+
}
|
|
5
484
|
}
|
metadata
CHANGED
|
@@ -1,57 +1,57 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: embulk-output-elasticsearch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Muga Nishizawa
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date:
|
|
11
|
+
date: 2016-01-26 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
|
-
name: bundler
|
|
15
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
16
|
-
requirements:
|
|
17
|
-
- - '>='
|
|
18
|
-
- !ruby/object:Gem::Version
|
|
19
|
-
version: '1.0'
|
|
20
14
|
requirement: !ruby/object:Gem::Requirement
|
|
21
15
|
requirements:
|
|
22
16
|
- - '>='
|
|
23
17
|
- !ruby/object:Gem::Version
|
|
24
18
|
version: '1.0'
|
|
19
|
+
name: bundler
|
|
25
20
|
prerelease: false
|
|
26
21
|
type: :development
|
|
27
|
-
- !ruby/object:Gem::Dependency
|
|
28
|
-
name: rake
|
|
29
22
|
version_requirements: !ruby/object:Gem::Requirement
|
|
30
23
|
requirements:
|
|
31
24
|
- - '>='
|
|
32
25
|
- !ruby/object:Gem::Version
|
|
33
|
-
version: '
|
|
26
|
+
version: '1.0'
|
|
27
|
+
- !ruby/object:Gem::Dependency
|
|
34
28
|
requirement: !ruby/object:Gem::Requirement
|
|
35
29
|
requirements:
|
|
36
30
|
- - '>='
|
|
37
31
|
- !ruby/object:Gem::Version
|
|
38
32
|
version: '10.0'
|
|
33
|
+
name: rake
|
|
39
34
|
prerelease: false
|
|
40
35
|
type: :development
|
|
41
|
-
- !ruby/object:Gem::Dependency
|
|
42
|
-
name: test-unit
|
|
43
36
|
version_requirements: !ruby/object:Gem::Requirement
|
|
44
37
|
requirements:
|
|
45
|
-
- -
|
|
38
|
+
- - '>='
|
|
46
39
|
- !ruby/object:Gem::Version
|
|
47
|
-
version:
|
|
40
|
+
version: '10.0'
|
|
41
|
+
- !ruby/object:Gem::Dependency
|
|
48
42
|
requirement: !ruby/object:Gem::Requirement
|
|
49
43
|
requirements:
|
|
50
44
|
- - ~>
|
|
51
45
|
- !ruby/object:Gem::Version
|
|
52
46
|
version: 3.0.2
|
|
47
|
+
name: test-unit
|
|
53
48
|
prerelease: false
|
|
54
49
|
type: :development
|
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
51
|
+
requirements:
|
|
52
|
+
- - ~>
|
|
53
|
+
- !ruby/object:Gem::Version
|
|
54
|
+
version: 3.0.2
|
|
55
55
|
description: Elasticsearch output plugin is an Embulk plugin that loads records to Elasticsearch read by any input plugins. Search the input plugins by "embulk-input" keyword.
|
|
56
56
|
email:
|
|
57
57
|
- muga.nishizawa@gmail.com
|
|
@@ -60,6 +60,7 @@ extensions: []
|
|
|
60
60
|
extra_rdoc_files: []
|
|
61
61
|
files:
|
|
62
62
|
- .gitignore
|
|
63
|
+
- CHANGELOG.md
|
|
63
64
|
- README.md
|
|
64
65
|
- build.gradle
|
|
65
66
|
- gradle/wrapper/gradle-wrapper.jar
|
|
@@ -70,24 +71,35 @@ files:
|
|
|
70
71
|
- settings.gradle
|
|
71
72
|
- src/main/java/org/embulk/output/elasticsearch/ElasticsearchOutputPlugin.java
|
|
72
73
|
- src/test/java/org/embulk/output/elasticsearch/TestElasticsearchOutputPlugin.java
|
|
73
|
-
-
|
|
74
|
-
- classpath/
|
|
75
|
-
- classpath/
|
|
76
|
-
- classpath/elasticsearch-
|
|
77
|
-
- classpath/embulk-output-elasticsearch-0.
|
|
78
|
-
- classpath/
|
|
79
|
-
- classpath/
|
|
80
|
-
- classpath/
|
|
81
|
-
- classpath/
|
|
82
|
-
- classpath/
|
|
83
|
-
- classpath/
|
|
84
|
-
- classpath/
|
|
85
|
-
- classpath/
|
|
86
|
-
- classpath/lucene-
|
|
87
|
-
- classpath/lucene-
|
|
88
|
-
- classpath/lucene-
|
|
89
|
-
- classpath/lucene-
|
|
74
|
+
- src/test/resources/sample_01.csv
|
|
75
|
+
- classpath/commons-cli-1.3.1.jar
|
|
76
|
+
- classpath/compress-lzf-1.0.2.jar
|
|
77
|
+
- classpath/elasticsearch-2.0.0.jar
|
|
78
|
+
- classpath/embulk-output-elasticsearch-0.2.0.jar
|
|
79
|
+
- classpath/HdrHistogram-2.1.6.jar
|
|
80
|
+
- classpath/hppc-0.7.1.jar
|
|
81
|
+
- classpath/jackson-dataformat-cbor-2.5.3.jar
|
|
82
|
+
- classpath/jackson-dataformat-smile-2.5.3.jar
|
|
83
|
+
- classpath/jackson-dataformat-yaml-2.5.3.jar
|
|
84
|
+
- classpath/joda-convert-1.2.jar
|
|
85
|
+
- classpath/joda-time-2.8.2.jar
|
|
86
|
+
- classpath/jsr166e-1.1.0.jar
|
|
87
|
+
- classpath/lucene-analyzers-common-5.2.1.jar
|
|
88
|
+
- classpath/lucene-backward-codecs-5.2.1.jar
|
|
89
|
+
- classpath/lucene-core-5.2.1.jar
|
|
90
|
+
- classpath/lucene-grouping-5.2.1.jar
|
|
91
|
+
- classpath/lucene-highlighter-5.2.1.jar
|
|
92
|
+
- classpath/lucene-join-5.2.1.jar
|
|
93
|
+
- classpath/lucene-memory-5.2.1.jar
|
|
94
|
+
- classpath/lucene-misc-5.2.1.jar
|
|
95
|
+
- classpath/lucene-queries-5.2.1.jar
|
|
96
|
+
- classpath/lucene-queryparser-5.2.1.jar
|
|
97
|
+
- classpath/lucene-sandbox-5.2.1.jar
|
|
98
|
+
- classpath/lucene-spatial-5.2.1.jar
|
|
99
|
+
- classpath/lucene-suggest-5.2.1.jar
|
|
100
|
+
- classpath/netty-3.10.5.Final.jar
|
|
90
101
|
- classpath/spatial4j-0.4.1.jar
|
|
102
|
+
- classpath/t-digest-3.0.jar
|
|
91
103
|
homepage: https://github.com/muga/embulk-output-elasticsearch
|
|
92
104
|
licenses:
|
|
93
105
|
- Apache 2.0
|