fluent-plugin-dynamodb 0.1.5 → 0.1.6
Sign up to get free protection for your applications and to get access to all the features.
- data/AUTHORS +3 -0
- data/README.md +70 -0
- data/VERSION +1 -1
- data/lib/fluent/plugin/out_dynamodb.rb +18 -18
- metadata +46 -53
data/AUTHORS
ADDED
data/README.md
CHANGED
@@ -32,6 +32,76 @@ Specify table name, hash attribute name and throughput as you like. fluent-plugi
|
|
32
32
|
* **dynamo\_db\_endpoint (required)** - end point of dynamodb. see [Regions and Endpoints](http://docs.amazonwebservices.com/general/latest/gr/rande.html#ddb_region)
|
33
33
|
* **dynamo\_db\_table (required)** - table name of dynamodb.
|
34
34
|
|
35
|
+
##TIPS
|
36
|
+
|
37
|
+
###retrieving data
|
38
|
+
|
39
|
+
fluent-plugin-dynamo will add **time** attribute and any other attributes of record automatically.
|
40
|
+
For example if you read apache's access log via fluentd, structure of the table will have been like this.
|
41
|
+
|
42
|
+
<table>
|
43
|
+
<tr>
|
44
|
+
<th>id (Hash Key)</th>
|
45
|
+
<th>time</th>
|
46
|
+
<th>host</th>
|
47
|
+
<th>path</th>
|
48
|
+
<th>method</th>
|
49
|
+
<th>referer</th>
|
50
|
+
<th>code</th>
|
51
|
+
<th>agent</th>
|
52
|
+
<th>size</th>
|
53
|
+
</tr>
|
54
|
+
<tr>
|
55
|
+
<td>"a937f980-b304-11e1-bc96-c82a14fffef2"</td>
|
56
|
+
<td>"2012-06-10T05:26:46Z"</td>
|
57
|
+
<td>"192.168.0.3"</td>
|
58
|
+
<td>"/index.html"</td>
|
59
|
+
<td>"GET"</td>
|
60
|
+
<td>"-"</td>
|
61
|
+
<td>"200"</td>
|
62
|
+
<td>"Mozilla/5.0"</td>
|
63
|
+
<td>"4286"</td>
|
64
|
+
</tr>
|
65
|
+
<tr>
|
66
|
+
<td>"a87fc51e-b308-11e1-ba0f-5855caf50759"</td>
|
67
|
+
<td>"2012-06-10T05:28:23Z"</td>
|
68
|
+
<td>"192.168.0.4"</td>
|
69
|
+
<td>"/sample.html"</td>
|
70
|
+
<td>"GET"</td>
|
71
|
+
<td>"-"</td>
|
72
|
+
<td>"200"</td>
|
73
|
+
<td>"Mozilla/5.0"</td>
|
74
|
+
<td>"8933"</td>
|
75
|
+
</tr>
|
76
|
+
</table>
|
77
|
+
|
78
|
+
Item can be retrieved by the key, but fluent-plugin-dynamo using UUID as a primary key.
|
79
|
+
There is no simple way to retrieve logs you want.
|
80
|
+
By the way, you can write scan-filter with AWS SDK like [this](https://gist.github.com/2906291), but Hive on EMR is the best practice I think.
|
81
|
+
|
82
|
+
###multi-region redundancy
|
83
|
+
|
84
|
+
As you know fluentd has **copy** output plugin.
|
85
|
+
So you can easily setup multi-region redundancy as follows.
|
86
|
+
|
87
|
+
<match dynamo.**>
|
88
|
+
type copy
|
89
|
+
<store>
|
90
|
+
type dynamodb
|
91
|
+
aws_key_id AWS_ACCESS_KEY
|
92
|
+
aws_sec_key AWS_SECRET_ACCESS_KEY
|
93
|
+
dynamo_db_table test
|
94
|
+
dynamo_db_endpoint dynamodb.ap-northeast-1.amazonaws.com
|
95
|
+
</store>
|
96
|
+
<store>
|
97
|
+
type dynamodb
|
98
|
+
aws_key_id AWS_ACCESS_KEY
|
99
|
+
aws_sec_key AWS_SECRET_ACCESS_KEY
|
100
|
+
dynamo_db_table test
|
101
|
+
dynamo_db_endpoint dynamodb.ap-southeast-1.amazonaws.com
|
102
|
+
</store>
|
103
|
+
</match>
|
104
|
+
|
35
105
|
##TODO
|
36
106
|
|
37
107
|
* auto-create table
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.1.
|
1
|
+
0.1.6
|
@@ -68,31 +68,31 @@ class DynamoDBOutput < Fluent::BufferedOutput
|
|
68
68
|
record[@hash_key_value] = UUIDTools::UUID.timestamp_create.to_s
|
69
69
|
end
|
70
70
|
|
71
|
-
[time
|
71
|
+
record['time'] = @timef.format(time)
|
72
|
+
|
73
|
+
record.to_msgpack
|
72
74
|
end
|
73
75
|
|
74
76
|
def write(chunk)
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
if
|
81
|
-
|
82
|
-
|
77
|
+
batch_size = 0
|
78
|
+
batch_records = []
|
79
|
+
chunk.msgpack_each {|record|
|
80
|
+
batch_records << record
|
81
|
+
batch_size += record.to_json.length # FIXME: heuristic
|
82
|
+
if batch_records.size >= BATCHWRITE_ITEM_LIMIT || batch_size >= BATCHWRITE_CONTENT_SIZE_LIMIT
|
83
|
+
batch_put_records(batch_records)
|
84
|
+
batch_records.clear
|
85
|
+
batch_size = 0
|
83
86
|
end
|
84
|
-
@batch.put(@dynamo_db_table, [record])
|
85
87
|
}
|
86
|
-
|
88
|
+
unless batch_records.empty?
|
89
|
+
batch_put_records(batch_records)
|
90
|
+
end
|
87
91
|
end
|
88
92
|
|
89
|
-
def
|
90
|
-
records
|
91
|
-
|
92
|
-
record['time'] = @timef.format(time)
|
93
|
-
records << record
|
94
|
-
}
|
95
|
-
records
|
93
|
+
def batch_put_records(records)
|
94
|
+
@batch.put(@dynamo_db_table, records)
|
95
|
+
@batch.process!
|
96
96
|
end
|
97
97
|
|
98
98
|
end
|
metadata
CHANGED
@@ -1,70 +1,67 @@
|
|
1
|
-
--- !ruby/object:Gem::Specification
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-dynamodb
|
3
|
-
version: !ruby/object:Gem::Version
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.6
|
4
5
|
prerelease:
|
5
|
-
version: 0.1.5
|
6
6
|
platform: ruby
|
7
|
-
authors:
|
7
|
+
authors:
|
8
8
|
- Takashi Matsuno
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
- !ruby/object:Gem::Dependency
|
12
|
+
date: 2012-06-12 00:00:00.000000000Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
16
15
|
name: fluentd
|
17
|
-
|
18
|
-
requirement: &id001 !ruby/object:Gem::Requirement
|
16
|
+
requirement: &16493400 !ruby/object:Gem::Requirement
|
19
17
|
none: false
|
20
|
-
requirements:
|
18
|
+
requirements:
|
21
19
|
- - ~>
|
22
|
-
- !ruby/object:Gem::Version
|
20
|
+
- !ruby/object:Gem::Version
|
23
21
|
version: 0.10.0
|
24
22
|
type: :runtime
|
25
|
-
version_requirements: *id001
|
26
|
-
- !ruby/object:Gem::Dependency
|
27
|
-
name: aws-sdk
|
28
23
|
prerelease: false
|
29
|
-
|
24
|
+
version_requirements: *16493400
|
25
|
+
- !ruby/object:Gem::Dependency
|
26
|
+
name: aws-sdk
|
27
|
+
requirement: &16492920 !ruby/object:Gem::Requirement
|
30
28
|
none: false
|
31
|
-
requirements:
|
29
|
+
requirements:
|
32
30
|
- - ~>
|
33
|
-
- !ruby/object:Gem::Version
|
31
|
+
- !ruby/object:Gem::Version
|
34
32
|
version: 1.5.2
|
35
33
|
type: :runtime
|
36
|
-
version_requirements: *id002
|
37
|
-
- !ruby/object:Gem::Dependency
|
38
|
-
name: uuidtools
|
39
34
|
prerelease: false
|
40
|
-
|
35
|
+
version_requirements: *16492920
|
36
|
+
- !ruby/object:Gem::Dependency
|
37
|
+
name: uuidtools
|
38
|
+
requirement: &16492460 !ruby/object:Gem::Requirement
|
41
39
|
none: false
|
42
|
-
requirements:
|
40
|
+
requirements:
|
43
41
|
- - ~>
|
44
|
-
- !ruby/object:Gem::Version
|
42
|
+
- !ruby/object:Gem::Version
|
45
43
|
version: 2.1.0
|
46
44
|
type: :runtime
|
47
|
-
version_requirements: *id003
|
48
|
-
- !ruby/object:Gem::Dependency
|
49
|
-
name: rake
|
50
45
|
prerelease: false
|
51
|
-
|
46
|
+
version_requirements: *16492460
|
47
|
+
- !ruby/object:Gem::Dependency
|
48
|
+
name: rake
|
49
|
+
requirement: &16492000 !ruby/object:Gem::Requirement
|
52
50
|
none: false
|
53
|
-
requirements:
|
54
|
-
- -
|
55
|
-
- !ruby/object:Gem::Version
|
51
|
+
requirements:
|
52
|
+
- - ! '>='
|
53
|
+
- !ruby/object:Gem::Version
|
56
54
|
version: 0.9.2
|
57
55
|
type: :development
|
58
|
-
|
56
|
+
prerelease: false
|
57
|
+
version_requirements: *16492000
|
59
58
|
description: Amazon DynamoDB output plugin for Fluent event collector
|
60
59
|
email: g0n5uk3@gmail.com
|
61
60
|
executables: []
|
62
|
-
|
63
61
|
extensions: []
|
64
|
-
|
65
62
|
extra_rdoc_files: []
|
66
|
-
|
67
|
-
|
63
|
+
files:
|
64
|
+
- AUTHORS
|
68
65
|
- ChangeLog
|
69
66
|
- Gemfile
|
70
67
|
- README.md
|
@@ -75,30 +72,26 @@ files:
|
|
75
72
|
- test/out_dynamodb.rb
|
76
73
|
homepage: https://github.com/fluent/fluent-plugin-dynamodb
|
77
74
|
licenses: []
|
78
|
-
|
79
75
|
post_install_message:
|
80
76
|
rdoc_options: []
|
81
|
-
|
82
|
-
require_paths:
|
77
|
+
require_paths:
|
83
78
|
- lib
|
84
|
-
required_ruby_version: !ruby/object:Gem::Requirement
|
79
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
85
80
|
none: false
|
86
|
-
requirements:
|
87
|
-
- -
|
88
|
-
- !ruby/object:Gem::Version
|
89
|
-
version:
|
90
|
-
required_rubygems_version: !ruby/object:Gem::Requirement
|
81
|
+
requirements:
|
82
|
+
- - ! '>='
|
83
|
+
- !ruby/object:Gem::Version
|
84
|
+
version: '0'
|
85
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
91
86
|
none: false
|
92
|
-
requirements:
|
93
|
-
- -
|
94
|
-
- !ruby/object:Gem::Version
|
95
|
-
version:
|
87
|
+
requirements:
|
88
|
+
- - ! '>='
|
89
|
+
- !ruby/object:Gem::Version
|
90
|
+
version: '0'
|
96
91
|
requirements: []
|
97
|
-
|
98
92
|
rubyforge_project:
|
99
|
-
rubygems_version: 1.8.
|
93
|
+
rubygems_version: 1.8.17
|
100
94
|
signing_key:
|
101
95
|
specification_version: 3
|
102
96
|
summary: Amazon DynamoDB output plugin for Fluent event collector
|
103
|
-
test_files:
|
104
|
-
- test/out_dynamodb.rb
|
97
|
+
test_files: []
|