fluent-plugin-aliyun-odps 0.1.0 → 0.1.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +3 -1
- data/README.cn.md +8 -13
- data/README.md +8 -21
- data/VERSION +1 -1
- data/lib/fluent/plugin/out_aliyun_odps.rb +12 -10
- data/lib/fluent/plugin/serialize/serializer.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 96472dd18671b2adc39de0934ea13aac995b81b9
|
4
|
+
data.tar.gz: b9f1ffc43b9f1b81be6081201d811074e28a3bb5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e3d190f5bbc2f7efb676485541b593b7a5e58177aa9ac41a6c04ba4f958594e3540a57c3fdd04571e5c6ee4e765cfe8419fc8265302f2bb1e4c103ee8f0ec4e0
|
7
|
+
data.tar.gz: f179cef524c9925924238c6660d05df73eaaa10833ac228b59d326f07cbd3b5b3be69885f71e5b4c1d3d3436997451a48db113b824ab3c953ff77004bba4e85e
|
data/CHANGELOG.md
CHANGED
data/README.cn.md
CHANGED
@@ -27,18 +27,6 @@
|
|
27
27
|
$ gem install fluent-plugin-aliyun-odps
|
28
28
|
```
|
29
29
|
|
30
|
-
### Դ�밲װ���
|
31
|
-
|
32
|
-
* ���Ȱ�װ����������fluentd�Լ�protobuf��
|
33
|
-
* Ȼ���Ŀ¼������Fluentd���Ŀ¼�£����磺 ��aliyun-odps-fluentd-plugin/lib/fluent/pluginĿ¼���Ƶ�{YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin�С�
|
34
|
-
|
35
|
-
```
|
36
|
-
$ gem install protobuf
|
37
|
-
$ gem install fluentd --no-ri --no-rdoc
|
38
|
-
$ git clone https://github.com/aliyun/aliyun-odps-fluentd-plugin.git
|
39
|
-
$ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin/ -r
|
40
|
-
```
|
41
|
-
|
42
30
|
### ���ʹ��ʾ��
|
43
31
|
|
44
32
|
```
|
@@ -63,12 +51,16 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
|
|
63
51
|
buffer_queue_limit 128
|
64
52
|
flush_interval 5s
|
65
53
|
project your_projectName
|
54
|
+
enable_fast_crc true
|
66
55
|
<table in.log>
|
67
56
|
table your_tableName
|
68
57
|
fields remote,method,path,code,size,agent
|
69
58
|
partition ctime=${datetime.strftime('%Y%m%d')}
|
70
59
|
time_format %d/%b/%Y:%H:%M:%S %z
|
71
60
|
shard_number 1
|
61
|
+
retry_time 3
|
62
|
+
retry_interval 1
|
63
|
+
abandon_mode true
|
72
64
|
</table>
|
73
65
|
</match>
|
74
66
|
```
|
@@ -94,7 +86,10 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
|
|
94
86
|
- time_format(Optional):
|
95
87
|
- ���ʹ��ʱ���ʽ�ؼ���Ϊ<partition>, �����ñ�����. ����: source[datetime]="29/Aug/2015:11:10:16 +0800",������<time_format>Ϊ"%d/%b/%Y:%H:%M:%S %z"
|
96
88
|
- shard_number(Optional):ָ��shard���������������shard[0,shard_number-1]��Χ�ڵ�shard��д�����ݣ�����Ϊ����0��С��table��Ӧshard������������.
|
97
|
-
- enable_fast_crc(Optional): ʹ�ÿ���crc���㣬�⽫�����������ܣ���������ʹ�����ⲿ���صĶ�̬���ӿ⣬Ŀǰ��֧��64λlinux��windows
|
89
|
+
- enable_fast_crc(Optional): ʹ�ÿ���crc���㣬�⽫�����������ܣ���������ʹ�����ⲿ���صĶ�̬���ӿ⣬Ŀǰ��֧��64λlinux��windowsϵͳ.
|
90
|
+
- retry_time(Optional): ����ÿ��pack����ʱ�������Դ�����Ĭ��3��.
|
91
|
+
- retry_interval(Optional): ���Լ����Ĭ��1s.
|
92
|
+
- abandon_mode(Optional): Ĭ��Ϊfalse�����ó�true��������retry_time�����������ݰ�������Ὣ�쳣����fluentd������fluentd�����Ի������ԣ�����������ܻᵼ�������ظ�.
|
98
93
|
|
99
94
|
## �ٷ���վ
|
100
95
|
---
|
data/README.md
CHANGED
@@ -20,30 +20,10 @@ To get started using this plugin, you will need these things:
|
|
20
20
|
|
21
21
|
### Install the Plugin
|
22
22
|
|
23
|
-
install the
|
23
|
+
install the plugin from gem:
|
24
24
|
|
25
25
|
```
|
26
26
|
$ gem install fluent-plugin-aliyun-odps
|
27
|
-
$ git clone https://github.com/aliyun/aliyun-odps-fluentd-plugin.git
|
28
|
-
```
|
29
|
-
|
30
|
-
Use gem to install dependency:
|
31
|
-
|
32
|
-
```
|
33
|
-
$ gem install protobuf
|
34
|
-
$ gem install fluentd --no-ri --no-rdoc
|
35
|
-
```
|
36
|
-
|
37
|
-
Your plugin is in aliyun-odps-fluentd-plugin/lib/fluent/plugin, entry file is out_odps.rb.
|
38
|
-
|
39
|
-
### Use the Plugin
|
40
|
-
|
41
|
-
- If you installed this plugin from gem, please ignore this step.
|
42
|
-
- Move the plugin dir into the plugin directory of Fluentd.
|
43
|
-
- (i.e., copy the folder aliyun-odps-fluentd-plugin/lib/fluent/plugin into {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin).
|
44
|
-
|
45
|
-
```
|
46
|
-
$ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin/ -r
|
47
27
|
```
|
48
28
|
|
49
29
|
### ODPS Fluentd plugin now is available. Following is a simple example of how to write ODPS output configuration.
|
@@ -70,12 +50,16 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
|
|
70
50
|
buffer_queue_limit 128
|
71
51
|
flush_interval 5s
|
72
52
|
project your_projectName
|
53
|
+
enable_fast_crc true
|
73
54
|
<table in.log>
|
74
55
|
table your_tableName
|
75
56
|
fields remote,method,path,code,size,agent
|
76
57
|
partition ctime=${datetime.strftime('%Y%m%d')}
|
77
58
|
time_format %d/%b/%Y:%H:%M:%S %z
|
78
59
|
shard_number 1
|
60
|
+
retry_time 3
|
61
|
+
retry_interval 1
|
62
|
+
abandon_mode true
|
79
63
|
</table>
|
80
64
|
</match>
|
81
65
|
```
|
@@ -101,6 +85,9 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
|
|
101
85
|
- if you are using the key words to set your <partition> and the key word is in time format, please set the param <time_format>. example: source[datetime] = "29/Aug/2015:11:10:16 +0800", and the param <time_format> is "%d/%b/%Y:%H:%M:%S %z"
|
102
86
|
- shard_number(Optional): will write data to shards between [0,shard_number-1], this config must more than 0 and less than the max shard number of your table.
|
103
87
|
- enable_fast_crc(Optional): use fast crc.so to calculate crc, this will improve speed up a lot, but this is not supported in some os.
|
88
|
+
- retry_time(Optional): retry times when exception happens for each pack, default 3.
|
89
|
+
- retry_interval(Optional): interval for retry, default 1s.
|
90
|
+
- abandon_mode(Optional): default false. Setting this to true will abandon pack data after @retry_time, otherwise will raise a exception to fluentd and use fluentd's retry, but this may cause duplicated data.
|
104
91
|
|
105
92
|
## Useful Links
|
106
93
|
---
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.1.
|
1
|
+
0.1.1
|
@@ -134,7 +134,7 @@ module Fluent
|
|
134
134
|
partition_name+=","+partition_column+"="+partition_value
|
135
135
|
end
|
136
136
|
else
|
137
|
-
raise "partition has no corresponding source key or the partition expression is wrong,"+data.
|
137
|
+
raise "partition has no corresponding source key or the partition expression is wrong,"+data.to_s
|
138
138
|
end
|
139
139
|
elsif p.include? "=${"
|
140
140
|
key=p[p.index("{")+1, p.index("}")-1-p.index("{")]
|
@@ -190,29 +190,30 @@ module Fluent
|
|
190
190
|
restCount = v.size%@thread_number
|
191
191
|
end
|
192
192
|
@client.createStreamArrayWriter().write(v[sendCount*threadId..sendCount*(threadId+1)+restCount-1], k)
|
193
|
-
@log.info "Successfully
|
193
|
+
@log.info "Successfully import "+(sendCount+restCount).to_s+" data to partition:"+k+",table:"+@table+" at threadId:"+threadId.to_s
|
194
194
|
rescue => e
|
195
|
+
@log.warn "Fail to write, error at threadId:"+threadId.to_s+" Msg:"+e.message + " partitions:" + k.to_s + " table:" + @table
|
195
196
|
# reload shard
|
196
197
|
if e.message.include? "ShardNotReady" or e.message.include? "InvalidShardId"
|
197
|
-
@log.warn "
|
198
|
+
@log.warn "Reload shard."
|
198
199
|
@client.loadShard(@shard_number)
|
199
200
|
@client.waitForShardLoad
|
200
201
|
elsif e.message.include? "NoSuchPartition"
|
201
202
|
begin
|
202
203
|
@client.addPartition(k)
|
203
|
-
@log.info "
|
204
|
+
@log.info "Add partition "+ k + " table:" + @table
|
204
205
|
rescue => ex
|
205
|
-
@log.error "
|
206
|
+
@log.error "Add partition failed"+ ex.message + " partitions:" + k.to_s + " table:" + @table
|
206
207
|
end
|
207
208
|
end
|
208
209
|
if retryTime > 0
|
209
|
-
@log.
|
210
|
+
@log.info "Retry in " + @retry_interval.to_s + "sec. Partitions:" + k.to_s + " table:" + @table
|
210
211
|
sleep(@retry_interval)
|
211
212
|
retryTime -= 1
|
212
213
|
retry
|
213
214
|
else
|
214
215
|
if (@abandon_mode)
|
215
|
-
@log.error "Retry failed, abandon this pack. Msg:" + e.message
|
216
|
+
@log.error "Retry failed, abandon this pack. Msg:" + e.message + " partitions:" + k.to_s + " table:" + @table
|
216
217
|
else
|
217
218
|
raise e
|
218
219
|
end
|
@@ -236,20 +237,21 @@ module Fluent
|
|
236
237
|
@client.createStreamArrayWriter().write(records[sendCount*threadId..sendCount*(threadId+1)+restCount-1])
|
237
238
|
@log.info "Successfully import "+(sendCount+restCount).to_s+" data to table:"+@table+" at threadId:"+threadId.to_s
|
238
239
|
rescue => e
|
240
|
+
@log.warn "Fail to write, error at threadId:"+threadId.to_s+" Msg:"+e.message + " table:" + @table
|
239
241
|
# reload shard
|
240
242
|
if e.message.include? "ShardNotReady" or e.message.include? "InvalidShardId"
|
241
|
-
@log.warn "
|
243
|
+
@log.warn "Reload shard."
|
242
244
|
@client.loadShard(@shard_number)
|
243
245
|
@client.waitForShardLoad
|
244
246
|
end
|
245
247
|
if retryTime > 0
|
246
|
-
@log.
|
248
|
+
@log.info "Retry in " + @retry_interval.to_s + "sec. Table:" + @table
|
247
249
|
sleep(@retry_interval)
|
248
250
|
retryTime -= 1
|
249
251
|
retry
|
250
252
|
else
|
251
253
|
if (@abandon_mode)
|
252
|
-
@log.error "Retry failed, abandon this pack. Msg:" + e.message
|
254
|
+
@log.error "Retry failed, abandon this pack. Msg:" + e.message + " Table:" + @table
|
253
255
|
else
|
254
256
|
raise e
|
255
257
|
end
|
@@ -62,7 +62,7 @@ module OdpsDatahub
|
|
62
62
|
|
63
63
|
def encodeString(value)
|
64
64
|
value_to_encode = value.dup
|
65
|
-
|
65
|
+
value_to_encode.encode!(::Protobuf::Field::StringField::ENCODING, :invalid => :replace, :undef => :replace, :replace => "")
|
66
66
|
value_to_encode.force_encoding(::Protobuf::Field::BytesField::BYTES_ENCODING)
|
67
67
|
string_bytes = ::Protobuf::Field::VarintField.encode(value_to_encode.size)
|
68
68
|
string_bytes << value_to_encode
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-aliyun-odps
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Xiao Dong
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2016-01-
|
12
|
+
date: 2016-01-22 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: fluentd
|