fluent-plugin-aliyun-odps 0.1.0 → 0.1.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (8) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +3 -1
  3. data/README.cn.md +8 -13
  4. data/README.md +8 -21
  5. data/VERSION +1 -1
  6. data/lib/fluent/plugin/out_aliyun_odps.rb +12 -10
  7. data/lib/fluent/plugin/serialize/serializer.rb +1 -1
  8. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 24f7ee66fa1af9233ffca9aa110d10c6e74cf4d7
4
- data.tar.gz: 7dd141195033863d501c3e2be9ecc0e0e2075c94
3
+ metadata.gz: 96472dd18671b2adc39de0934ea13aac995b81b9
4
+ data.tar.gz: b9f1ffc43b9f1b81be6081201d811074e28a3bb5
5
5
  SHA512:
6
- metadata.gz: 749bc9bfc987581ed7bbbff279ba61a695a3138ddb11a1f8c3d8d3588eee60d1c3c67a0aebb4fffa144215869827dcc9abb029551f6a9b87d75ebc55eaf4ea9e
7
- data.tar.gz: 87ab9ea47266abb85bf222195cd64210a6fd9284864cec71964bd405da8fcd53bcfa68cc1802615024642e97ea71281b205a0069c6a22f890eec972cd750c546
6
+ metadata.gz: e3d190f5bbc2f7efb676485541b593b7a5e58177aa9ac41a6c04ba4f958594e3540a57c3fdd04571e5c6ee4e765cfe8419fc8265302f2bb1e4c103ee8f0ec4e0
7
+ data.tar.gz: f179cef524c9925924238c6660d05df73eaaa10833ac228b59d326f07cbd3b5b3be69885f71e5b4c1d3d3436997451a48db113b824ab3c953ff77004bba4e85e
data/CHANGELOG.md CHANGED
@@ -11,4 +11,6 @@ Add abandon mode, fix fluent retry bug, fix partition mixed mode bug.
11
11
  0.0.9
12
12
  Hotfix retry log error bug.
13
13
  0.1.0
14
- Add partition when catch NoSuchPartition.
14
+ Add partition when catch NoSuchPartition.
15
+ 0.1.1
16
+ Fix some log format.
data/README.cn.md CHANGED
@@ -27,18 +27,6 @@
27
27
  $ gem install fluent-plugin-aliyun-odps
28
28
  ```
29
29
 
30
- ### Դ�밲װ���
31
-
32
- * ���Ȱ�װ����������fluentd�Լ�protobuf��
33
- * Ȼ�󽫲��Ŀ¼������Fluentd���Ŀ¼�£����磺 ��aliyun-odps-fluentd-plugin/lib/fluent/pluginĿ¼���Ƶ�{YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin�С�
34
-
35
- ```
36
- $ gem install protobuf
37
- $ gem install fluentd --no-ri --no-rdoc
38
- $ git clone https://github.com/aliyun/aliyun-odps-fluentd-plugin.git
39
- $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin/ -r
40
- ```
41
-
42
30
  ### ���ʹ��ʾ��
43
31
 
44
32
  ```
@@ -63,12 +51,16 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
63
51
  buffer_queue_limit 128
64
52
  flush_interval 5s
65
53
  project your_projectName
54
+ enable_fast_crc true
66
55
  <table in.log>
67
56
  table your_tableName
68
57
  fields remote,method,path,code,size,agent
69
58
  partition ctime=${datetime.strftime('%Y%m%d')}
70
59
  time_format %d/%b/%Y:%H:%M:%S %z
71
60
  shard_number 1
61
+ retry_time 3
62
+ retry_interval 1
63
+ abandon_mode true
72
64
  </table>
73
65
  </match>
74
66
  ```
@@ -94,7 +86,10 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
94
86
  - time_format(Optional):
95
87
  - ���ʹ��ʱ���ʽ�ؼ���Ϊ<partition>, �����ñ�����. ����: source[datetime]="29/Aug/2015:11:10:16 +0800",������<time_format>Ϊ"%d/%b/%Y:%H:%M:%S %z"
96
88
  - shard_number(Optional):ָ��shard���������������shard[0,shard_number-1]��Χ�ڵ�shard��д�����ݣ�����Ϊ����0��С��table��Ӧshard�������޵�����.
97
- - enable_fast_crc(Optional): ʹ�ÿ���crc���㣬�⽫�����������ܣ���������ʹ�����ⲿ���صĶ�̬���ӿ⣬Ŀǰ��֧��64λlinux��windowsϵͳ��
89
+ - enable_fast_crc(Optional): ʹ�ÿ���crc���㣬�⽫�����������ܣ���������ʹ�����ⲿ���صĶ�̬���ӿ⣬Ŀǰ��֧��64λlinux��windowsϵͳ.
90
+ - retry_time(Optional): ����ÿ��pack����ʱ�������Դ�����Ĭ��3��.
91
+ - retry_interval(Optional): ���Լ����Ĭ��1s.
92
+ - abandon_mode(Optional): Ĭ��Ϊfalse�����ó�true��������retry_time�����������ݰ�������Ὣ�쳣���͸�fluentd������fluentd�����Ի������ԣ�����������ܻᵼ�������ظ�.
98
93
 
99
94
  ## �ٷ���վ
100
95
  ---
data/README.md CHANGED
@@ -20,30 +20,10 @@ To get started using this plugin, you will need these things:
20
20
 
21
21
  ### Install the Plugin
22
22
 
23
- install the project from gem or github:
23
+ install the plugin from gem:
24
24
 
25
25
  ```
26
26
  $ gem install fluent-plugin-aliyun-odps
27
- $ git clone https://github.com/aliyun/aliyun-odps-fluentd-plugin.git
28
- ```
29
-
30
- Use gem to install dependency:
31
-
32
- ```
33
- $ gem install protobuf
34
- $ gem install fluentd --no-ri --no-rdoc
35
- ```
36
-
37
- Your plugin is in aliyun-odps-fluentd-plugin/lib/fluent/plugin, entry file is out_odps.rb.
38
-
39
- ### Use the Plugin
40
-
41
- - If you installed this plugin from gem, please ignore this step.
42
- - Move the plugin dir into the plugin directory of Fluentd.
43
- - (i.e., copy the folder aliyun-odps-fluentd-plugin/lib/fluent/plugin into {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin).
44
-
45
- ```
46
- $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin/ -r
47
27
  ```
48
28
 
49
29
  ### ODPS Fluentd plugin now is available. Following is a simple example of how to write ODPS output configuration.
@@ -70,12 +50,16 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
70
50
  buffer_queue_limit 128
71
51
  flush_interval 5s
72
52
  project your_projectName
53
+ enable_fast_crc true
73
54
  <table in.log>
74
55
  table your_tableName
75
56
  fields remote,method,path,code,size,agent
76
57
  partition ctime=${datetime.strftime('%Y%m%d')}
77
58
  time_format %d/%b/%Y:%H:%M:%S %z
78
59
  shard_number 1
60
+ retry_time 3
61
+ retry_interval 1
62
+ abandon_mode true
79
63
  </table>
80
64
  </match>
81
65
  ```
@@ -101,6 +85,9 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
101
85
  - if you are using the key words to set your <partition> and the key word is in time format, please set the param <time_format>. example: source[datetime] = "29/Aug/2015:11:10:16 +0800", and the param <time_format> is "%d/%b/%Y:%H:%M:%S %z"
102
86
  - shard_number(Optional): will write data to shards between [0,shard_number-1], this config must more than 0 and less than the max shard number of your table.
103
87
  - enable_fast_crc(Optional): use fast crc.so to calculate crc, this will improve speed up a lot, but this is not supported in some os.
88
+ - retry_time(Optional): retry times when exception happens for each pack, default 3.
89
+ - retry_interval(Optional): interval for retry, default 1s.
90
+ - abandon_mode(Optional): default false. Setting this to true will abandon pack data after @retry_time, otherwise will raise a exception to fluentd and use fluentd's retry, but this may cause duplicated data.
104
91
 
105
92
  ## Useful Links
106
93
  ---
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.1.0
1
+ 0.1.1
data/lib/fluent/plugin/out_aliyun_odps.rb CHANGED
@@ -134,7 +134,7 @@ module Fluent
134
134
  partition_name+=","+partition_column+"="+partition_value
135
135
  end
136
136
  else
137
- raise "partition has no corresponding source key or the partition expression is wrong,"+data.data.to_s
137
+ raise "partition has no corresponding source key or the partition expression is wrong,"+data.to_s
138
138
  end
139
139
  elsif p.include? "=${"
140
140
  key=p[p.index("{")+1, p.index("}")-1-p.index("{")]
@@ -190,29 +190,30 @@ module Fluent
190
190
  restCount = v.size%@thread_number
191
191
  end
192
192
  @client.createStreamArrayWriter().write(v[sendCount*threadId..sendCount*(threadId+1)+restCount-1], k)
193
- @log.info "Successfully import "+(sendCount+restCount).to_s+" data to partition:"+k+",table:"+@table+" at threadId:"+threadId.to_s
193
+ @log.info "Successfully import "+(sendCount+restCount).to_s+" data to partition:"+k+",table:"+@table+" at threadId:"+threadId.to_s
194
194
  rescue => e
195
+ @log.warn "Fail to write, error at threadId:"+threadId.to_s+" Msg:"+e.message + " partitions:" + k.to_s + " table:" + @table
195
196
  # reload shard
196
197
  if e.message.include? "ShardNotReady" or e.message.include? "InvalidShardId"
197
- @log.warn "write failed, msg:" + e.message + ", reload shard."
198
+ @log.warn "Reload shard."
198
199
  @client.loadShard(@shard_number)
199
200
  @client.waitForShardLoad
200
201
  elsif e.message.include? "NoSuchPartition"
201
202
  begin
202
203
  @client.addPartition(k)
203
- @log.info "add partition "+ k
204
+ @log.info "Add partition "+ k + " table:" + @table
204
205
  rescue => ex
205
- @log.error "add partition failed"+ ex.message
206
+ @log.error "Add partition failed"+ ex.message + " partitions:" + k.to_s + " table:" + @table
206
207
  end
207
208
  end
208
209
  if retryTime > 0
209
- @log.warn "Fail to write, retry in " + @retry_interval.to_s + "sec. Error at threadId:"+threadId.to_s+" Msg:"+e.message
210
+ @log.info "Retry in " + @retry_interval.to_s + "sec. Partitions:" + k.to_s + " table:" + @table
210
211
  sleep(@retry_interval)
211
212
  retryTime -= 1
212
213
  retry
213
214
  else
214
215
  if (@abandon_mode)
215
- @log.error "Retry failed, abandon this pack. Msg:" + e.message
216
+ @log.error "Retry failed, abandon this pack. Msg:" + e.message + " partitions:" + k.to_s + " table:" + @table
216
217
  else
217
218
  raise e
218
219
  end
@@ -236,20 +237,21 @@ module Fluent
236
237
  @client.createStreamArrayWriter().write(records[sendCount*threadId..sendCount*(threadId+1)+restCount-1])
237
238
  @log.info "Successfully import "+(sendCount+restCount).to_s+" data to table:"+@table+" at threadId:"+threadId.to_s
238
239
  rescue => e
240
+ @log.warn "Fail to write, error at threadId:"+threadId.to_s+" Msg:"+e.message + " table:" + @table
239
241
  # reload shard
240
242
  if e.message.include? "ShardNotReady" or e.message.include? "InvalidShardId"
241
- @log.warn "write failed, msg:" + e.message + ", reload shard."
243
+ @log.warn "Reload shard."
242
244
  @client.loadShard(@shard_number)
243
245
  @client.waitForShardLoad
244
246
  end
245
247
  if retryTime > 0
246
- @log.warn "Fail to write, retry in " + @retry_interval.to_s + "sec. Error at threadId:"+threadId.to_s+" Msg:"+e.message
248
+ @log.info "Retry in " + @retry_interval.to_s + "sec. Table:" + @table
247
249
  sleep(@retry_interval)
248
250
  retryTime -= 1
249
251
  retry
250
252
  else
251
253
  if (@abandon_mode)
252
- @log.error "Retry failed, abandon this pack. Msg:" + e.message
254
+ @log.error "Retry failed, abandon this pack. Msg:" + e.message + " Table:" + @table
253
255
  else
254
256
  raise e
255
257
  end
data/lib/fluent/plugin/serialize/serializer.rb CHANGED
@@ -62,7 +62,7 @@ module OdpsDatahub
62
62
 
63
63
  def encodeString(value)
64
64
  value_to_encode = value.dup
65
- #value_to_encode.encode!(::Protobuf::Field::StringField::ENCODING, :invalid => :replace, :undef => :replace, :replace => "")
65
+ value_to_encode.encode!(::Protobuf::Field::StringField::ENCODING, :invalid => :replace, :undef => :replace, :replace => "")
66
66
  value_to_encode.force_encoding(::Protobuf::Field::BytesField::BYTES_ENCODING)
67
67
  string_bytes = ::Protobuf::Field::VarintField.encode(value_to_encode.size)
68
68
  string_bytes << value_to_encode
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-aliyun-odps
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.1.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Xiao Dong
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2016-01-06 00:00:00.000000000 Z
12
+ date: 2016-01-22 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: fluentd