fluent-plugin-aliyun-odps 0.0.3 → 0.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 6111e8fe5ab6fe5c217be596ceb517cd5cf079d8
-  data.tar.gz: b2bb072a2c9055e8aad1b03df64410b4db00b5de
+  metadata.gz: 8f2a437d1d6538a45d143aee240a08c6026450e9
+  data.tar.gz: 66c71203d5721ce578511f531e6e118755bcdf0f
 SHA512:
-  metadata.gz: 1c5ddb658b5fa359a979fc75503ce44ae8ecf92574357bab81ea6b37295f50eacb57f3ace7ceda625bcc8881760517c2eb2860a8477a29e3f98ea2ae39d0468e
-  data.tar.gz: 90b5b780617c4c16a11618880f8f0a0954d8505caab08d3edf72ab29ee15bc7023b530f17bdb636a580769fc0dd3475a8865b7c1012b8a238f13596edfc85acf
+  metadata.gz: 01aa158831e7788e28b58d8765ebb138f6489a23b673b75e5546881f0ca9118090e67406c51bae7592bcf1b47e3377473ac558b18d46ce1b811474cf9d72f80d
+  data.tar.gz: e2e82e1492eb4de88e093acf83196ce3ec0a159d76657d5dd210d027c1364c0ccc00ca760ceedaadf28eb999ec1fe273f4d4df46cad257f35ed709401cef19b0

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+0.0.4
+Fix datetime format bug, support String, DateTime, Time type when write to a datetime field.

data/README.cn.md

@@ -0,0 +1,112 @@
+# Aliyun ODPS Plugin for Fluentd
+
+## ��ʼʹ��
+---
+
+### ����
+
+- �������ݴ�������(Open Data Processing Service�����ODPS)�ǰ���Ͱ������з��ĺ������ݴ���ƽ̨����Ҫ�����������ṹ�����ݵĴ洢�ͼ��㣬�����ṩ�������ݲֿ�Ľ�������Լ���Դ����ݵķ�����ģ����
+- ODPS DataHub Service(DHS)��һ��ODPS���ڽ��������û��ṩʵʱ���ݵķ���(Publish)�Ͷ���(Subscribe)�Ĺ��ܡ�
+
+
+### ����Ҫ��
+
+ʹ�ô˲������Ҫ�߱����»���:
+
+1. Ruby 2.1.0 �����
+2. Gem 2.4.5 �����
+3. Fluentd-0.10.49 ����� (*[Home Page](http://www.fluentd.org/)*)
+4. Protobuf-3.5.1 �����(Ruby protobuf)
+
+### GEM��װ
+
+��ruby gem��װʹ��:
+
+```
+$ gem install fluent-plugin-aliyun-odps
+```
+
+### Դ�밲װ���
+
+* ���Ȱ�װ����������fluentd�Լ�protobuf��
+* Ȼ�󽫲��Ŀ¼������Fluentd���Ŀ¼�£����磺 ��aliyun-odps-fluentd-plugin/lib/fluent/pluginĿ¼���Ƶ�{YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin�С�
+
+```
+$ gem install protobuf
+$ gem install fluentd --no-ri --no-rdoc
+$ git clone https://github.com/aliyun/aliyun-odps-fluentd-plugin.git
+$ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib/fluent/plugin/ -r
+```
+
+### ���ʹ��ʾ��
+
+```
+<source>
+   type tail
+   path /opt/log/in/in.log
+   pos_file /opt/log/in/in.log.pos
+   refresh_interval 5s
+   tag in.log
+   format /^(?<remote>[^ ]*) - - \[(?<datetime>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "-" "(?<agent>[^\"]*)"$/
+   time_format %Y%b%d %H:%M:%S %z
+</source>
+```
+```
+<match in.**>
+  type aliyun_odps
+  aliyun_access_id ************
+  aliyun_access_key *********
+  aliyun_odps_endpoint http://service.odps.aliyun.com/api
+  aliyun_odps_hub_endpoint http://dh.odps.aliyun.com
+  buffer_chunk_limit 2m
+  buffer_queue_limit 128
+  flush_interval 5s
+  project your_projectName
+  <table in.log>
+	table your_tableName
+	fields remote,method,path,code,size,agent
+	partition ctime=${datetime.strftime('%Y%m%d')}
+	time_format %d/%b/%Y:%H:%M:%S %z
+	shard_number 1
+  </table>
+</match>
+```
+### ����˵��
+
+- type(Fixed): �̶�ֵ aliyun_odps.
+- aliyun_access_id(Required):������access_id.
+- aliyun_access_key(Required):������access key.
+- aliyun_odps_hub_endpoint(Required):�����ķ�������ESC�ϣ���ѱ�ֵ�趨Ϊ http://dh-ext.odps.aliyun-inc.com, ��������Ϊ http://dh.odps.aliyun.com.
+- aliyunodps_endpoint(Required):�����ķ�������ESC�ϣ���ѱ�ֵ�趨Ϊ http://odps-ext.aiyun-inc.com/api, ��������Ϊ http://service.odps.aliyun.com/api .
+- buffer_chunk_limit(Optional): ���С��֧�֡�k��(KB),��m��(MB),��g��(GB)��λ��Ĭ�� 8MB������ֵ2MB.
+- buffer_queue_limit(Optional): ����д�С����ֵ��buffer_chunk_limit��ͬ����������������С��
+- flush_interval(Optional): ǿ�Ʒ��ͼ�����ﵽʱ��������δ����ǿ�Ʒ���, Ĭ�� 60s.
+- project(Required): project����.
+- table(Required): table����.
+- fields(Required): ��source��Ӧ���ֶ������������source֮��.
+- partition(Optional)����Ϊ�������������ô���.
+    - ������֧�ֵ�����ģʽ:
+        - �̶�ֵ: partition ctime=20150804
+        - �ؼ���: partition ctime=${remote} ������remoteΪsource��ij�ֶΣ�
+        - ʱ���ʽ�ؼ���: partition ctime=${datetime.strftime('%Y%m%d')} ������datetimeΪsource��ijʱ���ʽ�ֶΣ����Ϊ%Y%m%d��ʽ��Ϊ�������ƣ�
+- time_format(Optional):
+    - ���ʹ��ʱ���ʽ�ؼ���Ϊ<partition>, �����ñ�����. ����: source[datetime]="29/Aug/2015:11:10:16 +0800",������<time_format>Ϊ"%d/%b/%Y:%H:%M:%S %z"
+- shard_number(Optional):ָ��shard����������С��create tableʱָ����shardֵ����.
+
+## �ٷ���վ
+---
+
+- [Fluentd User Guide](http://docs.fluentd.org/)
+
+## ����
+---
+
+- [Sun Zongtao]()
+- [Cai Ying]()
+- [Dong Xiao](https://github.com/dongxiao1198)
+- [Yang Hongbo](https://github.com/hongbosoftware)
+
+## License
+---
+
+licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)

data/README.md

@@ -8,10 +8,9 @@
 - ODPS-Open Data Processing Service is a massive data processing platform designed by alibaba.
 - DHS-ODPS DataHub Service is a service in Odps, which provides real-time upload and download functions for user.
 
-
 ### Requirements
 
-To get started using this plugin, you will need three things:
+To get started using this plugin, you will need these things:
 
 1. Ruby 2.1.0 or later
 2. Gem 2.4.5 or later
@@ -97,10 +96,9 @@ $ cp aliyun-odps-fluentd-plugin/lib/fluent/plugin/* {YOUR_FLUENTD_DIRECTORY}/lib
         - key words: partition ctime=${remote}
         - key words int time format: partition ctime=${datetime.strftime('%Y%m%d')}
 - time_format(Optional):
-    - if the odps table is partitioned, you need to set the param <partition>
     - if you are using the key words to set your <partition> and the key word is in time format, please set the param <time_format>. example: source[datetime] = "29/Aug/2015:11:10:16 +0800", and the param <time_format> is "%d/%b/%Y:%H:%M:%S %z"
 - shard_number(Optional):less than the number you set when create the hubtable.
-  
+
 ## Useful Links
 ---
 

data/VERSION

@@ -1 +1 @@
-0.0.3
+0.0.4

data/lib/fluent/plugin/odps/odps_table.rb

@@ -77,10 +77,23 @@ module OdpsDatahub
     end
 
     def setDateTime(idx, value)
-      if value.is_a?Integer and value >= $DATETIME_MIN_TICKS and  value <=> $DATETIME_MAX_TICKS
+      if value.is_a?Integer and value >= $DATETIME_MIN_TICKS and  value <= $DATETIME_MAX_TICKS
         setValue(idx, value)
+      elsif value.is_a?DateTime or value.is_a?Time
+        if value.to_i*1000 >= $DATETIME_MIN_TICKS and  value.to_i*1000 <= $DATETIME_MAX_TICKS
+          setValue(idx, value.to_i*1000)
+        else
+          raise "DateTime out of range or value show be Integer and between -62135798400000 and 253402271999000."
+        end
+      elsif value.is_a?String
+        begin
+          tmpTime = Time.parse(value)
+          setValue(idx, tmpTime.to_i*1000)
+        rescue
+          raise "Parse string to datetime failed, string:" + value
+        end
       else
-        raise "DateTime out of range or value show be Integer"
+        raise "DateTime cell should be in Integer or Time or DateTime format."
       end
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-aliyun-odps
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.0.4
 platform: ruby
 authors:
 - Xiao Dong
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-09-07 00:00:00.000000000 Z
+date: 2015-09-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -125,6 +125,7 @@ files:
 - CHANGELOG.md
 - Gemfile
 - License
+- README.cn.md
 - README.md
 - Rakefile
 - VERSION