fluent-plugin-aliyun-odps 0.0.3 → 0.0.4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +2 -0
- data/README.cn.md +112 -0
- data/README.md +2 -4
- data/VERSION +1 -1
- data/lib/fluent/plugin/odps/odps_table.rb +15 -2
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 8f2a437d1d6538a45d143aee240a08c6026450e9
|
4
|
+
data.tar.gz: 66c71203d5721ce578511f531e6e118755bcdf0f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 01aa158831e7788e28b58d8765ebb138f6489a23b673b75e5546881f0ca9118090e67406c51bae7592bcf1b47e3377473ac558b18d46ce1b811474cf9d72f80d
|
7
|
+
data.tar.gz: e2e82e1492eb4de88e093acf83196ce3ec0a159d76657d5dd210d027c1364c0ccc00ca760ceedaadf28eb999ec1fe273f4d4df46cad257f35ed709401cef19b0
|
data/CHANGELOG.md
CHANGED
data/README.cn.md
ADDED
@@ -0,0 +1,112 @@
|
|
1
|
+
# Aliyun ODPS Plugin for Fluentd
|
2
|
+
|
3
|
+
## ��ʼʹ��
|
4
|
+
---
|
5
|
+
|
6
|
+
### ����
|
7
|
+
|
8
|
+
- �������ݴ�������(Open Data Processing Service�����ODPS)�ǰ���Ͱ������з��ĺ������ݴ���ƽ̨����Ҫ�����������ṹ�����ݵĴ洢�ͼ��㣬�����ṩ�������ݲֿ�Ľ�������Լ���Դ����ݵķ�����ģ����
|
9
|
+
- ODPS DataHub Service(DHS)��һ��ODPS���ڽ��������û��ṩʵʱ���ݵķ���(Publish)�Ͷ���(Subscribe)�Ĺ��ܡ�
|
10
|
+
|
11
|
+
|
12
|
+
### ����Ҫ��
|
13
|
+
|
14
|
+
ʹ�ô˲������Ҫ�߱����»���:
|
15
|
+
|
16
|
+
1. Ruby 2.1.0 �����
|
17
|
+
2. Gem 2.4.5 �����
|
18
|
+
3. Fluentd-0.10.49 ����� (*[Home Page](http://www.fluentd.org/)*)
|
19
|
+
4. Protobuf-3.5.1 �����(Ruby protobuf)
|
20
|
+
|
21
|
+
### GEM��װ
|
22
|
+
|
23
|
+
��ruby gem��װʹ��:
|
24
|
+
|
25
|
+
```
|
26
|
+
$ gem install fluent-plugin-aliyun-odps
|
27
|
+
```
|
28
|
+
|
29
|
+
### Դ�밲װ���
|
30
|
+
|
31
|
+
* ���Ȱ�װ����������fluentd�Լ�protobuf��
|
32
|
+
* Ȼ���Ŀ¼������Fluentd���Ŀ¼�£����磺 ��aliyun-odps-fluentd-plugin/lib/fluent/pluginĿ¼���Ƶ�{YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin�С�
|
33
|
+
|
34
|
+
```
|
35
|
+
$ gem install protobuf
|
36
|
+
$ gem install fluentd --no-ri --no-rdoc
|
37
|
+
$ git clone https://github.com/aliyun/aliyun-odps-fluentd-plugin.git
|
38
|
+
$ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin/ -r
|
39
|
+
```
|
40
|
+
|
41
|
+
### ���ʹ��ʾ��
|
42
|
+
|
43
|
+
```
|
44
|
+
<source>
|
45
|
+
type tail
|
46
|
+
path /opt/log/in/in.log
|
47
|
+
pos_file /opt/log/in/in.log.pos
|
48
|
+
refresh_interval 5s
|
49
|
+
tag in.log
|
50
|
+
format /^(?<remote>[^ ]*) - - \[(?<datetime>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "-" "(?<agent>[^\"]*)"$/
|
51
|
+
time_format %Y%b%d %H:%M:%S %z
|
52
|
+
</source>
|
53
|
+
```
|
54
|
+
```
|
55
|
+
<match in.**>
|
56
|
+
type aliyun_odps
|
57
|
+
aliyun_access_id ************
|
58
|
+
aliyun_access_key *********
|
59
|
+
aliyun_odps_endpoint http://service.odps.aliyun.com/api
|
60
|
+
aliyun_odps_hub_endpoint http://dh.odps.aliyun.com
|
61
|
+
buffer_chunk_limit 2m
|
62
|
+
buffer_queue_limit 128
|
63
|
+
flush_interval 5s
|
64
|
+
project your_projectName
|
65
|
+
<table in.log>
|
66
|
+
table your_tableName
|
67
|
+
fields remote,method,path,code,size,agent
|
68
|
+
partition ctime=${datetime.strftime('%Y%m%d')}
|
69
|
+
time_format %d/%b/%Y:%H:%M:%S %z
|
70
|
+
shard_number 1
|
71
|
+
</table>
|
72
|
+
</match>
|
73
|
+
```
|
74
|
+
### ����˵��
|
75
|
+
|
76
|
+
- type(Fixed): �̶�ֵ aliyun_odps.
|
77
|
+
- aliyun_access_id(Required):������access_id.
|
78
|
+
- aliyun_access_key(Required):������access key.
|
79
|
+
- aliyun_odps_hub_endpoint(Required):�����ķ�������ESC�ϣ���ѱ�ֵ�趨Ϊ http://dh-ext.odps.aliyun-inc.com, ��������Ϊ http://dh.odps.aliyun.com.
|
80
|
+
- aliyunodps_endpoint(Required):�����ķ�������ESC�ϣ���ѱ�ֵ�趨Ϊ http://odps-ext.aiyun-inc.com/api, ��������Ϊ http://service.odps.aliyun.com/api .
|
81
|
+
- buffer_chunk_limit(Optional): ���С��֧�֡�k��(KB),��m��(MB),��g��(GB)��λ��Ĭ�� 8MB������ֵ2MB.
|
82
|
+
- buffer_queue_limit(Optional): ����д�С����ֵ��buffer_chunk_limit��ͬ����������������С��
|
83
|
+
- flush_interval(Optional): ǿ�Ʒ��ͼ�����ﵽʱ��������δ����ǿ�Ʒ���, Ĭ�� 60s.
|
84
|
+
- project(Required): project����.
|
85
|
+
- table(Required): table����.
|
86
|
+
- fields(Required): ��source��Ӧ���ֶ������������source֮��.
|
87
|
+
- partition(Optional)����Ϊ�������������ô���.
|
88
|
+
- ������֧�ֵ�����ģʽ:
|
89
|
+
- �̶�ֵ: partition ctime=20150804
|
90
|
+
- �ؼ���: partition ctime=${remote} ������remoteΪsource��ij�ֶΣ�
|
91
|
+
- ʱ���ʽ�ؼ���: partition ctime=${datetime.strftime('%Y%m%d')} ������datetimeΪsource��ijʱ���ʽ�ֶΣ����Ϊ%Y%m%d��ʽ��Ϊ�������ƣ�
|
92
|
+
- time_format(Optional):
|
93
|
+
- ���ʹ��ʱ���ʽ�ؼ���Ϊ<partition>, �����ñ�����. ����: source[datetime]="29/Aug/2015:11:10:16 +0800",������<time_format>Ϊ"%d/%b/%Y:%H:%M:%S %z"
|
94
|
+
- shard_number(Optional):ָ��shard����������С��create tableʱָ����shardֵ����.
|
95
|
+
|
96
|
+
## �ٷ���վ
|
97
|
+
---
|
98
|
+
|
99
|
+
- [Fluentd User Guide](http://docs.fluentd.org/)
|
100
|
+
|
101
|
+
## ����
|
102
|
+
---
|
103
|
+
|
104
|
+
- [Sun Zongtao]()
|
105
|
+
- [Cai Ying]()
|
106
|
+
- [Dong Xiao](https://github.com/dongxiao1198)
|
107
|
+
- [Yang Hongbo](https://github.com/hongbosoftware)
|
108
|
+
|
109
|
+
## License
|
110
|
+
---
|
111
|
+
|
112
|
+
licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)
|
data/README.md
CHANGED
@@ -8,10 +8,9 @@
|
|
8
8
|
- ODPS-Open Data Processing Service is a massive data processing platform designed by alibaba.
|
9
9
|
- DHS-ODPS DataHub Service is a service in Odps, which provides real-time upload and download functions for user.
|
10
10
|
|
11
|
-
|
12
11
|
### Requirements
|
13
12
|
|
14
|
-
To get started using this plugin, you will need
|
13
|
+
To get started using this plugin, you will need these things:
|
15
14
|
|
16
15
|
1. Ruby 2.1.0 or later
|
17
16
|
2. Gem 2.4.5 or later
|
@@ -97,10 +96,9 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
|
|
97
96
|
- key words: partition ctime=${remote}
|
98
97
|
- key words int time format: partition ctime=${datetime.strftime('%Y%m%d')}
|
99
98
|
- time_format(Optional):
|
100
|
-
- if the odps table is partitioned, you need to set the param <partition>
|
101
99
|
- if you are using the key words to set your <partition> and the key word is in time format, please set the param <time_format>. example: source[datetime] = "29/Aug/2015:11:10:16 +0800", and the param <time_format> is "%d/%b/%Y:%H:%M:%S %z"
|
102
100
|
- shard_number(Optional):less than the number you set when create the hubtable.
|
103
|
-
|
101
|
+
|
104
102
|
## Useful Links
|
105
103
|
---
|
106
104
|
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.0.
|
1
|
+
0.0.4
|
@@ -77,10 +77,23 @@ module OdpsDatahub
|
|
77
77
|
end
|
78
78
|
|
79
79
|
def setDateTime(idx, value)
|
80
|
-
if value.is_a?Integer and value >= $DATETIME_MIN_TICKS and value
|
80
|
+
if value.is_a?Integer and value >= $DATETIME_MIN_TICKS and value <= $DATETIME_MAX_TICKS
|
81
81
|
setValue(idx, value)
|
82
|
+
elsif value.is_a?DateTime or value.is_a?Time
|
83
|
+
if value.to_i*1000 >= $DATETIME_MIN_TICKS and value.to_i*1000 <= $DATETIME_MAX_TICKS
|
84
|
+
setValue(idx, value.to_i*1000)
|
85
|
+
else
|
86
|
+
raise "DateTime out of range or value show be Integer and between -62135798400000 and 253402271999000."
|
87
|
+
end
|
88
|
+
elsif value.is_a?String
|
89
|
+
begin
|
90
|
+
tmpTime = Time.parse(value)
|
91
|
+
setValue(idx, tmpTime.to_i*1000)
|
92
|
+
rescue
|
93
|
+
raise "Parse string to datetime failed, string:" + value
|
94
|
+
end
|
82
95
|
else
|
83
|
-
raise "DateTime
|
96
|
+
raise "DateTime cell should be in Integer or Time or DateTime format."
|
84
97
|
end
|
85
98
|
end
|
86
99
|
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-aliyun-odps
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.4
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Xiao Dong
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2015-09-
|
12
|
+
date: 2015-09-30 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: fluentd
|
@@ -125,6 +125,7 @@ files:
|
|
125
125
|
- CHANGELOG.md
|
126
126
|
- Gemfile
|
127
127
|
- License
|
128
|
+
- README.cn.md
|
128
129
|
- README.md
|
129
130
|
- Rakefile
|
130
131
|
- VERSION
|