fluent-plugin-azurestorage 0.0.7 → 0.0.8

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 920277cb0b8ab63f16a0626a8c1e9ea341e62b28
4
- data.tar.gz: 0a86562c7713af3c45f8863a503e29bdb5aed1cc
3
+ metadata.gz: 776476b29eb416de1bf483aaf7f2a7a864ed10fe
4
+ data.tar.gz: 95cdf6fa62d1f4e772179694a1a3f7a611ee3149
5
5
  SHA512:
6
- metadata.gz: 61f809feb8edb5810251a3ee39daeb6a169b4178e932bb02c7ceadc821755ac0bfb20c0a4cc2b3ea4ba4e7e77606c8cccb4b06aafc64cbc0cc4cf07dc65f25a4
7
- data.tar.gz: 477d61ae9f6faad494a1ef3d169929ad4416fa09ca7af21c450f70e0eaa70b1c3e85b9d3cff68dfa431ef204dac3216165e7691704bc571da99f0ce6a3c461d1
6
+ metadata.gz: cec4a7f65d722e23bb49e46211c482b10f7c3e9f15e0ed286b960599aac7034a5d110db122ab072ace1ea056a820b68d7e2f55a5b9033f6252a6ba71710718cc
7
+ data.tar.gz: b21756a7d13066e1155853ee3ed2bd9262ef24304483f0306c509f34538ae4ee9029efd0a1d161432f65520785c30e7033f35aeee9e10a68acaca710bc0cd6ad
data/.gitignore CHANGED
@@ -1,11 +1,11 @@
1
- ~*
2
- #*
3
- *~
4
- [._]*.s[a-w][a-z]
5
- .DS_Store
6
-
7
- *.gem
8
- .bundle
9
- Gemfile.lock
10
- vendor
11
- .ruby-version
1
+ ~*
2
+ #*
3
+ *~
4
+ [._]*.s[a-w][a-z]
5
+ .DS_Store
6
+
7
+ *.gem
8
+ .bundle
9
+ Gemfile.lock
10
+ vendor
11
+ .ruby-version
data/.travis.yml CHANGED
@@ -1,21 +1,19 @@
1
- language: ruby
2
-
3
- rvm:
4
- - 1.9.3
5
- - 2.0
6
- - 2.1
7
- - 2.2.0
8
- - rbx
9
-
10
- gemfile:
11
- - Gemfile
12
-
13
- branches:
14
- only:
15
- - master
16
-
17
- script: bundle exec rake test
18
-
19
- matrix:
20
- allow_failures:
21
- - rvm: rbx
1
+ language: ruby
2
+
3
+ rvm:
4
+ - 2.1
5
+ - 2.2.0
6
+ - rbx
7
+
8
+ gemfile:
9
+ - Gemfile
10
+
11
+ branches:
12
+ only:
13
+ - master
14
+
15
+ script: bundle exec rake test
16
+
17
+ matrix:
18
+ allow_failures:
19
+ - rvm: rbx
data/Gemfile CHANGED
@@ -1,3 +1,3 @@
1
- source "http://rubygems.org"
2
-
3
- gemspec
1
+ source "http://rubygems.org"
2
+
3
+ gemspec
data/LICENSE.txt CHANGED
@@ -1,22 +1,22 @@
1
- Copyright (c) 2014 Hidemasa Togashi
2
-
3
- MIT License
4
-
5
- Permission is hereby granted, free of charge, to any person obtaining
6
- a copy of this software and associated documentation files (the
7
- "Software"), to deal in the Software without restriction, including
8
- without limitation the rights to use, copy, modify, merge, publish,
9
- distribute, sublicense, and/or sell copies of the Software, and to
10
- permit persons to whom the Software is furnished to do so, subject to
11
- the following conditions:
12
-
13
- The above copyright notice and this permission notice shall be
14
- included in all copies or substantial portions of the Software.
15
-
16
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
- EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
- MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
- NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
- LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
- OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
- WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
1
+ Copyright (c) 2014 Hidemasa Togashi
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.md CHANGED
@@ -1,210 +1,210 @@
1
- # Azure Storage output plugin for Fluentd
2
-
3
- [![Build Status](https://travis-ci.org/htgc/fluent-plugin-azurestorage.svg?branch=master)](https://travis-ci.org/htgc/fluent-plugin-azurestorage)
4
-
5
- ## Overview
6
-
7
- Azure Storate output plugin buffers logs in local file and upload them to Azure Storage periodically.
8
- This plugin is porting from [fluent-plugin-s3](https://github.com/fluent/fluent-plugin-s3) for AzureStorage.
9
-
10
- ## Installation
11
-
12
- Install from RubyGems:
13
- ```
14
- $ gem install fluent-plugin-azurestorage
15
- ```
16
-
17
- ## Configuration
18
-
19
- ```
20
- <match pattern>
21
- type azurestorage
22
-
23
- azure_storage_account <your azure storage account>
24
- azure_storage_access_key <your azure storage access key>
25
- azure_container <your azure storage container>
26
- azure_storage_type blob
27
- store_as gzip
28
- auto_create_container true
29
- path logs/
30
- azure_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
31
- buffer_path /var/log/fluent/azurestorage
32
-
33
- time_slice_format %Y%m%d-%H
34
- time_slice_wait 10m
35
- utc
36
- </match>
37
- ```
38
-
39
- ### azure_storage_account
40
-
41
- Your Azure Storage Account Name. This can be got from Azure Management potal.
42
- This parameter is required when environment variable 'AZURE_STORAGE_ACCOUNT' is not set.
43
-
44
- ### azure_storage_access_key
45
-
46
- Your Azure Storage Access Key(Primary or Secondary). This also can be got from Azure Management potal.
47
- This parameter is required when environment variable 'AZURE_STORAGE_ACCESS_KEY' is not set.
48
-
49
- ### azure_container (Required)
50
-
51
- Azure Storage Container name
52
-
53
- ### auto_create_container
54
-
55
- This plugin create container if not exist when you set 'auto_create_container' to true.
56
-
57
- ### azure_storage_type
58
-
59
- Azure Storage type. Now supports only 'blob'(default). 'tables' and 'queues' are not implemented.
60
-
61
- ### azure_object_key_format
62
-
63
- The format of Azure Storage object keys. You can use several built-in variables:
64
-
65
- - %{path}
66
- - %{time_slice}
67
- - %{index}
68
- - %{file_extension}
69
-
70
- to decide keys dynamically.
71
-
72
- %{path} is exactly the value of *path* configured in the configuration file. E.g., "logs/" in the example configuration above.
73
- %{time_slice} is the time-slice in text that are formatted with *time_slice_format*.
74
- %{index} is the sequential number starts from 0, increments when multiple files are uploaded to Azure Storage in the same time slice.
75
- %{file_extention} is always "gz" for now.
76
-
77
- The default format is "%{path}%{time_slice}_%{index}.%{file_extension}".
78
-
79
- For instance, using the example configuration above, actual object keys on Azure Storage will be something like:
80
-
81
- ```
82
- "logs/20130111-22_0.gz"
83
- "logs/20130111-23_0.gz"
84
- "logs/20130111-23_1.gz"
85
- "logs/20130112-00_0.gz"
86
- ```
87
-
88
- With the configuration:
89
-
90
- ```
91
- azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
92
- path log
93
- time_slice_format %Y%m%d-%H
94
- ```
95
-
96
- You get:
97
-
98
- ```
99
- "log/events/ts=20130111-22/events_0.gz"
100
- "log/events/ts=20130111-23/events_0.gz"
101
- "log/events/ts=20130111-23/events_1.gz"
102
- "log/events/ts=20130112-00/events_0.gz"
103
- ```
104
-
105
- The [fluent-mixin-config-placeholders](https://github.com/tagomoris/fluent-mixin-config-placeholders) mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the azure_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.
106
-
107
- ```
108
- azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
109
- ```
110
-
111
- ### store_as
112
-
113
- Archive format on Azure Storage. You can use following types:
114
-
115
- - gzip (default)
116
- - json
117
- - text
118
- - lzo (Need lzop command)
119
- - lzma2 (Need xz command)
120
-
121
- ### format
122
-
123
- Change one line format in the Azure Storage object. Supported formats are 'out_file', 'json', 'ltsv' and 'single_value'.
124
-
125
- - out_file (default)
126
-
127
- ```
128
- time\ttag\t{..json1..}
129
- time\ttag\t{..json2..}
130
- ...
131
- ```
132
-
133
- - json
134
-
135
- ```
136
- {..json1..}
137
- {..json2..}
138
- ...
139
- ```
140
-
141
- At this format, "time" and "tag" are omitted.
142
- But you can set these information to the record by setting "include_tag_key" / "tag_key" and "include_time_key" / "time_key" option.
143
- If you set following configuration in AzureStorage output:
144
-
145
- ```
146
- format json
147
- include_time_key true
148
- time_key log_time # default is time
149
- ```
150
-
151
- then the record has log_time field.
152
-
153
- ```
154
- {"log_time":"time string",...}
155
- ```
156
-
157
- - ltsv
158
-
159
- ```
160
- key1:value1\tkey2:value2
161
- key1:value1\tkey2:value2
162
- ...
163
- ```
164
-
165
- "ltsv" format also accepts "include_xxx" related options. See "json" section.
166
-
167
- - single_value
168
-
169
- Use specified value instead of entire recode. If you get '{"message":"my log"}', then contents are
170
-
171
- ```
172
- my log1
173
- my log2
174
- ...
175
- ```
176
-
177
- You can change key name by "message_key" option.
178
-
179
- ### path
180
-
181
- path prefix of the files on Azure Storage. Default is "" (no prefix).
182
-
183
- ### buffer_path (required)
184
-
185
- path prefix of the files to buffer logs.
186
-
187
- ### time_slice_format
188
-
189
- Format of the time used as the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to split files hourly.
190
-
191
- ### time_slice_wait
192
-
193
- The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.
194
-
195
- ### utc
196
-
197
- Use UTC instead of local time.
198
-
199
- ## License
200
- Azure Storage output plugin is licensed according to the terms of the Apache License, Version 2.0.
201
-
202
- The full version of this lisence can be found at http://www.apache.org/licenses/LICENSE-2.0 [TXT](http://www.apache.org/licenses/LICENSE-2.0.txt) or [HTML](http://www.apache.org/licenses/LICENSE-2.0.html)
203
-
204
- ## Contributing
205
-
206
- 1. Fork it
207
- 2. Create your feature branch (`git checkout -b my-new-feature`)
208
- 3. Commit your changes (`git commit -am 'Add some feature'`)
209
- 4. Push to the branch (`git push origin my-new-feature`)
210
- 5. Create new Pull Request
1
+ # Azure Storage output plugin for Fluentd
2
+
3
+ [![Build Status](https://travis-ci.org/htgc/fluent-plugin-azurestorage.svg?branch=master)](https://travis-ci.org/htgc/fluent-plugin-azurestorage)
4
+
5
+ ## Overview
6
+
7
+ Azure Storate output plugin buffers logs in local file and upload them to Azure Storage periodically.
8
+ This plugin is porting from [fluent-plugin-s3](https://github.com/fluent/fluent-plugin-s3) for AzureStorage.
9
+
10
+ ## Installation
11
+
12
+ Install from RubyGems:
13
+ ```
14
+ $ gem install fluent-plugin-azurestorage
15
+ ```
16
+
17
+ ## Configuration
18
+
19
+ ```
20
+ <match pattern>
21
+ type azurestorage
22
+
23
+ azure_storage_account <your azure storage account>
24
+ azure_storage_access_key <your azure storage access key>
25
+ azure_container <your azure storage container>
26
+ azure_storage_type blob
27
+ store_as gzip
28
+ auto_create_container true
29
+ path logs/
30
+ azure_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
31
+ buffer_path /var/log/fluent/azurestorage
32
+
33
+ time_slice_format %Y%m%d-%H
34
+ time_slice_wait 10m
35
+ utc
36
+ </match>
37
+ ```
38
+
39
+ ### azure_storage_account
40
+
41
+ Your Azure Storage Account Name. This can be got from Azure Management potal.
42
+ This parameter is required when environment variable 'AZURE_STORAGE_ACCOUNT' is not set.
43
+
44
+ ### azure_storage_access_key
45
+
46
+ Your Azure Storage Access Key(Primary or Secondary). This also can be got from Azure Management potal.
47
+ This parameter is required when environment variable 'AZURE_STORAGE_ACCESS_KEY' is not set.
48
+
49
+ ### azure_container (Required)
50
+
51
+ Azure Storage Container name
52
+
53
+ ### auto_create_container
54
+
55
+ This plugin create container if not exist when you set 'auto_create_container' to true.
56
+
57
+ ### azure_storage_type
58
+
59
+ Azure Storage type. Now supports only 'blob'(default). 'tables' and 'queues' are not implemented.
60
+
61
+ ### azure_object_key_format
62
+
63
+ The format of Azure Storage object keys. You can use several built-in variables:
64
+
65
+ - %{path}
66
+ - %{time_slice}
67
+ - %{index}
68
+ - %{file_extension}
69
+
70
+ to decide keys dynamically.
71
+
72
+ %{path} is exactly the value of *path* configured in the configuration file. E.g., "logs/" in the example configuration above.
73
+ %{time_slice} is the time-slice in text that are formatted with *time_slice_format*.
74
+ %{index} is the sequential number starts from 0, increments when multiple files are uploaded to Azure Storage in the same time slice.
75
+ %{file_extention} is always "gz" for now.
76
+
77
+ The default format is "%{path}%{time_slice}_%{index}.%{file_extension}".
78
+
79
+ For instance, using the example configuration above, actual object keys on Azure Storage will be something like:
80
+
81
+ ```
82
+ "logs/20130111-22_0.gz"
83
+ "logs/20130111-23_0.gz"
84
+ "logs/20130111-23_1.gz"
85
+ "logs/20130112-00_0.gz"
86
+ ```
87
+
88
+ With the configuration:
89
+
90
+ ```
91
+ azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
92
+ path log
93
+ time_slice_format %Y%m%d-%H
94
+ ```
95
+
96
+ You get:
97
+
98
+ ```
99
+ "log/events/ts=20130111-22/events_0.gz"
100
+ "log/events/ts=20130111-23/events_0.gz"
101
+ "log/events/ts=20130111-23/events_1.gz"
102
+ "log/events/ts=20130112-00/events_0.gz"
103
+ ```
104
+
105
+ The [fluent-mixin-config-placeholders](https://github.com/tagomoris/fluent-mixin-config-placeholders) mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the azure_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.
106
+
107
+ ```
108
+ azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
109
+ ```
110
+
111
+ ### store_as
112
+
113
+ Archive format on Azure Storage. You can use following types:
114
+
115
+ - gzip (default)
116
+ - json
117
+ - text
118
+ - lzo (Need lzop command)
119
+ - lzma2 (Need xz command)
120
+
121
+ ### format
122
+
123
+ Change one line format in the Azure Storage object. Supported formats are 'out_file', 'json', 'ltsv' and 'single_value'.
124
+
125
+ - out_file (default)
126
+
127
+ ```
128
+ time\ttag\t{..json1..}
129
+ time\ttag\t{..json2..}
130
+ ...
131
+ ```
132
+
133
+ - json
134
+
135
+ ```
136
+ {..json1..}
137
+ {..json2..}
138
+ ...
139
+ ```
140
+
141
+ At this format, "time" and "tag" are omitted.
142
+ But you can set these information to the record by setting "include_tag_key" / "tag_key" and "include_time_key" / "time_key" option.
143
+ If you set following configuration in AzureStorage output:
144
+
145
+ ```
146
+ format json
147
+ include_time_key true
148
+ time_key log_time # default is time
149
+ ```
150
+
151
+ then the record has log_time field.
152
+
153
+ ```
154
+ {"log_time":"time string",...}
155
+ ```
156
+
157
+ - ltsv
158
+
159
+ ```
160
+ key1:value1\tkey2:value2
161
+ key1:value1\tkey2:value2
162
+ ...
163
+ ```
164
+
165
+ "ltsv" format also accepts "include_xxx" related options. See "json" section.
166
+
167
+ - single_value
168
+
169
+ Use specified value instead of entire recode. If you get '{"message":"my log"}', then contents are
170
+
171
+ ```
172
+ my log1
173
+ my log2
174
+ ...
175
+ ```
176
+
177
+ You can change key name by "message_key" option.
178
+
179
+ ### path
180
+
181
+ path prefix of the files on Azure Storage. Default is "" (no prefix).
182
+
183
+ ### buffer_path (required)
184
+
185
+ path prefix of the files to buffer logs.
186
+
187
+ ### time_slice_format
188
+
189
+ Format of the time used as the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to split files hourly.
190
+
191
+ ### time_slice_wait
192
+
193
+ The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.
194
+
195
+ ### utc
196
+
197
+ Use UTC instead of local time.
198
+
199
+ ## License
200
+ Azure Storage output plugin is licensed according to the terms of the Apache License, Version 2.0.
201
+
202
+ The full version of this lisence can be found at http://www.apache.org/licenses/LICENSE-2.0 [TXT](http://www.apache.org/licenses/LICENSE-2.0.txt) or [HTML](http://www.apache.org/licenses/LICENSE-2.0.html)
203
+
204
+ ## Contributing
205
+
206
+ 1. Fork it
207
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
208
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
209
+ 4. Push to the branch (`git push origin my-new-feature`)
210
+ 5. Create new Pull Request