logstash-output-qingstor 0.3.2 → 0.3.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +15 -2
- data/README.md +18 -57
- data/README_zh_CN.md +33 -58
- data/lib/logstash/outputs/qingstor/rotation_policy.rb +6 -6
- data/logstash-output-qingstor.gemspec +3 -3
- data/spec/outputs/qingstor/qingstor_validator_spec.rb +55 -9
- data/spec/outputs/qingstor/rotation_policy_spec.rb +46 -55
- data/spec/outputs/qingstor/temporary_file_factory_spec.rb +9 -2
- data/spec/outputs/qingstor_spec.rb +2 -3
- metadata +9 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 515c9929fb2a985fa7e97b8775615a2a57fee0c7ba8d7a1cafeb89d309fa55cf
|
4
|
+
data.tar.gz: 81e0dea2bd38b84f353f51505699ced9630ead77318654621a749271bbc5ed15
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 9f5b3b2496db39fc228404ca1b6f25d7e0f4f55d4499f7276c26795be7e02c3f798d7654cf1a75759b324de3eba7d596fbfcb5df739fcd34a0c31bb666e74a51
|
7
|
+
data.tar.gz: 6157494e770ff5c82a4b803af6c840087b7612d4c3d54461788b115203495513156476fbc0d6493ebb1475065e4ab63db13d01157b6011213b3d19a40f09cbfc
|
data/CHANGELOG.md
CHANGED
@@ -1,4 +1,5 @@
|
|
1
|
-
|
1
|
+
## 0.3.2
|
2
|
+
- Fix the errors of unit conversion in rotation policy
|
2
3
|
|
3
4
|
## 0.3.1
|
4
5
|
- Refactor module: rotation policy
|
@@ -17,6 +18,18 @@
|
|
17
18
|
|
18
19
|
## 0.2.5
|
19
20
|
- Fix: remove uuid directories which cannotbe removed properly
|
20
|
-
|
21
|
+
|
22
|
+
## 0.2.4
|
23
|
+
- Fix uploading empty files after logstash shutdown
|
24
|
+
|
25
|
+
## 0.2.2
|
26
|
+
- Fix bug: uploading empty gzip file
|
27
|
+
|
28
|
+
## 0.2.1
|
29
|
+
- Fix the wrong unit of file_time
|
30
|
+
|
31
|
+
## 0.2.0
|
32
|
+
- Fix redundant newline
|
33
|
+
|
21
34
|
## 0.1.0
|
22
35
|
- Plugin created with the logstash plugin generator
|
data/README.md
CHANGED
@@ -1,29 +1,25 @@
|
|
1
1
|
# Logstash Output Plugin for QingStor
|
2
|
+
[![Build Status](https://travis-ci.org/yunify/logstash-output-qingstor.svg?branch=master)](https://travis-ci.org/yunify/logstash-output-qingstor) [![Gem Version](https://badge.fury.io/rb/logstash-output-qingstor.svg)](https://badge.fury.io/rb/logstash-output-qingstor.svg) [![License](http://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/yunify/logstash-output-qingstor/blob/master/LICENSE) [![README Chinese](https://img.shields.io/badge/README-%E4%B8%AD%E6%96%87-blue.svg)](/README_zh_CN.md)
|
2
3
|
|
3
|
-
|
4
|
+
This is a Logstash output plugin, it collects the outputs from logstash, and store them in [QingStor](https://www.qingcloud.com/products/storage#qingstor).
|
4
5
|
|
5
|
-
|
6
|
+
> Incompatible with Logstash version 5.5.x. Please use the least or previous releases to avoid crashing down errors, such as 6.0.0+ or 5.4.x.
|
6
7
|
|
7
|
-
|
8
|
-
|
9
|
-
For now, We've submitted this plugin to rubygems.org. Use the following command to install.
|
8
|
+
## How to use
|
9
|
+
This plugin has submitted to [rubygems.org](rubygems.org). Use the following command to install:
|
10
10
|
|
11
11
|
``` bash
|
12
12
|
$ bin/logstash-plugin install logstash-output-qingstor
|
13
13
|
```
|
14
14
|
|
15
|
-
If you have installed a previous release, please use the folliwing command to update
|
15
|
+
If you have installed a previous release, please use the folliwing command to update:
|
16
16
|
|
17
17
|
```bash
|
18
18
|
$ bin/logstash-plugin update logstash-output-qingstor
|
19
19
|
```
|
20
20
|
|
21
|
-
|
22
|
-
|
23
|
-
## 1. Configuration Guide
|
24
|
-
|
25
|
-
#### 1.1 Run in minimal Configuration Items
|
26
|
-
|
21
|
+
#### Run in minimal Configuration Items
|
22
|
+
Edit a conf file, fill `output` field with qingstor configurations.
|
27
23
|
```sh
|
28
24
|
output {
|
29
25
|
qingstor {
|
@@ -36,53 +32,18 @@ output {
|
|
36
32
|
|
37
33
|
```
|
38
34
|
|
39
|
-
More configuration details please refer to [common options](/docs/index.asciidoc).
|
40
|
-
|
41
|
-
## 2. Running your unpublished Plugin in Logstash
|
42
|
-
|
43
|
-
#### 2.1 Run in a local Logstash clone
|
44
|
-
|
45
|
-
- Edit Logstash `Gemfile` and add the local plugin path, for example:
|
46
|
-
-
|
47
|
-
``` ruby
|
48
|
-
$ gem "logstash-output-qingstor", :path => "/your/local/logstash-output-qingstor"
|
49
|
-
```
|
50
|
-
|
51
|
-
- Install plugin
|
52
|
-
-
|
53
|
-
``` bash
|
54
|
-
$ bin/logstash-plugin install --no-verify
|
55
|
-
```
|
56
|
-
- Run Logstash with your plugin
|
35
|
+
> More configuration details please refer to [common options](/docs/index.asciidoc).
|
57
36
|
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
} \
|
65
|
-
}"
|
66
|
-
```
|
67
|
-
|
68
|
-
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
|
69
|
-
|
70
|
-
#### 2.2 Run in an installed Logstash
|
71
|
-
|
72
|
-
You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
|
73
|
-
|
74
|
-
- Build your plugin gem
|
75
|
-
|
76
|
-
``` bash
|
77
|
-
$ gem build logstash-output-qingstor.gemspec
|
78
|
-
```
|
79
|
-
- Install the plugin from the Logstash home
|
80
|
-
|
81
|
-
``` bash
|
82
|
-
$ bin/logstash-plugin install /your/local/plugin/logstash-output-qingstor.gem
|
83
|
-
```
|
37
|
+
## Features ([CHANGELOG](./CHANGELOG.md))
|
38
|
+
- Support gzip compress.
|
39
|
+
- Restore uncomplete files.
|
40
|
+
- Server size encrption.
|
41
|
+
- Redirect QingStor hosts.
|
42
|
+
- Multipart uploading supports 500GB file.
|
84
43
|
|
85
|
-
|
44
|
+
## TODO
|
45
|
+
- Custom stored file name.
|
46
|
+
- Restore uncomplete multipart uploading
|
86
47
|
|
87
48
|
## Contributing
|
88
49
|
Please see [Contributing Guidelines](./CONTRIBUTING.md) of this project before submitting patches.
|
data/README_zh_CN.md
CHANGED
@@ -1,23 +1,29 @@
|
|
1
1
|
# Logstash Output Plugin for QingStor
|
2
2
|
|
3
|
-
[English](/README.md)
|
3
|
+
[![Build Status](https://travis-ci.org/yunify/logstash-output-qingstor.svg?branch=master)](https://travis-ci.org/yunify/logstash-output-qingstor) [![Gem Version](https://badge.fury.io/rb/logstash-output-qingstor.svg)](https://badge.fury.io/rb/logstash-output-qingstor.svg) [![License](http://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/yunify/logstash-output-qingstor/blob/master/LICENSE) [![README English](https://img.shields.io/badge/README-English-blue.svg)](/README.md)
|
4
4
|
|
5
|
-
|
6
|
-
详细功能参考下面配置说明.
|
5
|
+
这是适配了 [QingStor](https://www.qingcloud.com/products/storage#qingstor) 的 Logstash output 插件。通过本插件可以将 Logstash 的结果导出到 QingStor 对象存储中。
|
7
6
|
|
7
|
+
> 已知在 Logstash 5.5.x 版本中会崩溃,请使用最新或者之前的 Logstash 版本。例如 6.0 以上版本或者 5.4 版本。
|
8
|
+
|
9
|
+
## 安装
|
8
10
|
目前插件已经提交至 [RubyGems](https://rubygems.org), 使用以下命令安装:
|
9
11
|
|
10
12
|
``` bash
|
11
13
|
$ bin/logstash-plugin install logstash-output-qingstor
|
12
14
|
```
|
13
15
|
|
14
|
-
|
16
|
+
如果你安装过一个早期的版本,可以通过以下的命令来更新插件:
|
17
|
+
|
18
|
+
``` bash
|
19
|
+
$ bin/logstash-plugin update logstash-output-qingstor
|
20
|
+
```
|
15
21
|
|
16
|
-
##
|
22
|
+
## 配置说明
|
23
|
+
|
24
|
+
#### 最小运行配置
|
25
|
+
编辑一个 `*.conf` 文件或者使用 `-e` 参数直接输入配置, 最小运行配置至少需要以下三项
|
17
26
|
|
18
|
-
#### 1.1 最小运行配置
|
19
|
-
- 使用 `-f` 接受一个 `*.conf` 文件或者使用 `-e` 参数直接输入配置, 最小运行配置至少需要以下三项
|
20
|
-
-
|
21
27
|
``` bash
|
22
28
|
output {
|
23
29
|
qingstor {
|
@@ -29,7 +35,7 @@ output {
|
|
29
35
|
}
|
30
36
|
```
|
31
37
|
|
32
|
-
####
|
38
|
+
#### 其他可选参数说明
|
33
39
|
|
34
40
|
``` bash
|
35
41
|
output {
|
@@ -58,7 +64,7 @@ output {
|
|
58
64
|
# 默认: "size_and_time". 可选枚举值["size_and_time", "size", "time"].
|
59
65
|
rotation_strategy => "size_and_time"
|
60
66
|
|
61
|
-
# 配合"size_and_time", "size"的可选配置型, 单位 MB
|
67
|
+
# 配合"size_and_time", "size"的可选配置型, 单位 megabyte(MB)
|
62
68
|
# 默认: 5 (MB)
|
63
69
|
size_file => 5
|
64
70
|
|
@@ -70,7 +76,7 @@ output {
|
|
70
76
|
# 默认: "none". 可选枚举值: ["AES256", "none"]
|
71
77
|
server_side_encryption_algorithm => "AES256"
|
72
78
|
|
73
|
-
# 选用服务端文件加密时提供的秘钥, 秘钥要求32
|
79
|
+
# 选用服务端文件加密时提供的秘钥, 秘钥要求 32 byte(256 bit)
|
74
80
|
customer_key => "your_encryption_key"
|
75
81
|
|
76
82
|
# 宕机恢复, 启动logstash时, 自动上传目录下的遗留文件
|
@@ -80,50 +86,19 @@ output {
|
|
80
86
|
}
|
81
87
|
}
|
82
88
|
```
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
- 使用插件运行
|
101
|
-
-
|
102
|
-
``` bash
|
103
|
-
$ bin/logstash -e "output { \
|
104
|
-
qingstor { \
|
105
|
-
access_key_id => 'your_access_key_id' \
|
106
|
-
secret_access_key => 'your_secret_access_key' \
|
107
|
-
bucket => 'bucket_name' \
|
108
|
-
} \
|
109
|
-
}"
|
110
|
-
```
|
111
|
-
|
112
|
-
此时你对插件所做的任意的代码上的修改都会直接生效.
|
113
|
-
|
114
|
-
#### 2.2 安装一个本地插件然后运行
|
115
|
-
|
116
|
-
这一步你需要生成一个插件的 Gem 包, 然后通过 Logstash 来安装到 Logstash 的插件目录下
|
117
|
-
- 在项目目录下生成 Gem
|
118
|
-
|
119
|
-
``` bash
|
120
|
-
$ gem build logstash-output-qingstor.gemspec
|
121
|
-
```
|
122
|
-
|
123
|
-
- 在 Logstash 的目录下使用 `logstash-plugin` 安装
|
124
|
-
|
125
|
-
``` bash
|
126
|
-
$ bin/logstash-plugin install /your/local/plugin/logstash-output-qingstor.gem
|
127
|
-
```
|
128
|
-
|
129
|
-
- 安装完毕之后, 就可以使用 Logstash 运行开始测试了.
|
89
|
+
## 特性 ([CHANGELOG](./CHANGELOG.md))
|
90
|
+
- 支持 gzip 压缩。
|
91
|
+
- 恢复上次宕机后未上传完成的文件。
|
92
|
+
- 服务端加密(AES256)。
|
93
|
+
- 重定向指自建的 QingStor 服务器。
|
94
|
+
- 分段上传支持最大 500GB 的文件。
|
95
|
+
|
96
|
+
## TODO
|
97
|
+
- 自定义上传文件的名称。
|
98
|
+
- 恢复意外终止的分段上传。
|
99
|
+
|
100
|
+
## Contributing
|
101
|
+
Please see [Contributing Guidelines](./CONTRIBUTING.md) of this project before submitting patches.
|
102
|
+
|
103
|
+
## LICENSE
|
104
|
+
The Apache License (Version 2.0, January 2004).
|
@@ -28,7 +28,7 @@ module LogStash
|
|
28
28
|
end
|
29
29
|
|
30
30
|
class Time < Policy
|
31
|
-
def initialize(
|
31
|
+
def initialize(_, file_time)
|
32
32
|
@file_time = file_time
|
33
33
|
positive_check(@file_time)
|
34
34
|
end
|
@@ -36,10 +36,10 @@ module LogStash
|
|
36
36
|
def rotate?(file)
|
37
37
|
!file.empty? && (::Time.now - file.ctime) >= @file_time
|
38
38
|
end
|
39
|
-
end
|
39
|
+
end
|
40
40
|
|
41
41
|
class Size < Policy
|
42
|
-
def initialize(file_size,
|
42
|
+
def initialize(file_size, _)
|
43
43
|
@file_size = file_size
|
44
44
|
positive_check(@file_size)
|
45
45
|
end
|
@@ -59,16 +59,16 @@ module LogStash
|
|
59
59
|
|
60
60
|
def rotate?(file)
|
61
61
|
(!file.empty? && (::Time.now - file.ctime) >= @file_time) ||
|
62
|
-
|
62
|
+
(file.size >= @file_size)
|
63
63
|
end
|
64
64
|
end
|
65
65
|
|
66
66
|
def Policy(policy, file_size, file_time)
|
67
67
|
case policy
|
68
68
|
when Policy then policy
|
69
|
-
else
|
69
|
+
else
|
70
70
|
self.class.const_get(policy.to_s.split('_').map(&:capitalize).join)
|
71
|
-
|
71
|
+
.new(file_size, file_time)
|
72
72
|
end
|
73
73
|
end
|
74
74
|
|
@@ -1,7 +1,7 @@
|
|
1
1
|
Gem::Specification.new do |s|
|
2
2
|
s.name = 'logstash-output-qingstor'
|
3
|
-
s.version = '0.3.
|
4
|
-
s.licenses = ['Apache
|
3
|
+
s.version = '0.3.3'
|
4
|
+
s.licenses = ['Apache-2.0']
|
5
5
|
s.summary = 'logstash output plugin for qingstor'
|
6
6
|
s.description = 'Collect the outputs of logstash and store into QingStor'
|
7
7
|
s.homepage = 'https://github.com/yunify/logstash-output-qingstor'
|
@@ -20,7 +20,7 @@ Gem::Specification.new do |s|
|
|
20
20
|
# Gem dependencies
|
21
21
|
s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.0'
|
22
22
|
s.add_runtime_dependency 'logstash-codec-plain'
|
23
|
-
s.add_runtime_dependency 'qingstor-sdk', '>= 1.9.3'
|
23
|
+
s.add_runtime_dependency 'qingstor-sdk', '~> 1.9', '>= 1.9.3'
|
24
24
|
s.add_runtime_dependency 'concurrent-ruby'
|
25
25
|
|
26
26
|
s.add_development_dependency 'stud', '~> 0.0.22'
|
@@ -2,22 +2,68 @@
|
|
2
2
|
|
3
3
|
require 'logstash/devutils/rspec/spec_helper'
|
4
4
|
require 'logstash/outputs/qingstor/qingstor_validator'
|
5
|
-
require_relative '../qs_access_helper'
|
6
5
|
|
7
6
|
describe LogStash::Outputs::Qingstor::QingstorValidator do
|
7
|
+
def get_bucket(config)
|
8
|
+
access_key_id = config.fetch('access_key_id')
|
9
|
+
secret_access_key = config.fetch('secret_access_key')
|
10
|
+
bucket = config.fetch('bucket')
|
11
|
+
region = config.fetch('region')
|
12
|
+
config = QingStor::SDK::Config.init(access_key_id, secret_access_key)
|
13
|
+
properties = { 'bucket-name' => bucket, 'zone' => region }
|
14
|
+
QingStor::SDK::Bucket.new(config, properties)
|
15
|
+
end
|
16
|
+
|
8
17
|
let(:normal_prefix) { 'super/bucket' }
|
9
18
|
let(:wrong_prefix1) { '/wrong/prefix' }
|
10
19
|
let(:wrong_prefix2) { normal_prefix * 100 }
|
11
|
-
let(:
|
20
|
+
let(:config) do
|
21
|
+
{ 'access_key_id' => ENV['access_key_id'],
|
22
|
+
'secret_access_key' => ENV['secret_access_key'],
|
23
|
+
'bucket' => ENV['bucket'],
|
24
|
+
'region' => ENV['region'] }
|
25
|
+
end
|
26
|
+
|
27
|
+
context 'validate the prefix' do
|
28
|
+
it 'raise error if the prefix is not valid' do
|
29
|
+
expect { described_class.prefix_valid?(wrong_prefix1) }
|
30
|
+
.to raise_error(LogStash::ConfigurationError)
|
31
|
+
expect { described_class.prefix_valid?(wrong_prefix2) }
|
32
|
+
.to raise_error(LogStash::ConfigurationError)
|
33
|
+
end
|
12
34
|
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
expect { described_class.prefix_valid?(wrong_prefix2) }
|
17
|
-
.to raise_error(LogStash::ConfigurationError)
|
35
|
+
it 'return true if the prefix is valid' do
|
36
|
+
expect(described_class.prefix_valid?(normal_prefix)).to be_truthy
|
37
|
+
end
|
18
38
|
end
|
19
39
|
|
20
|
-
|
21
|
-
|
40
|
+
context 'validate the bucket' do
|
41
|
+
it 'tests with wrong id' do
|
42
|
+
config['access_key_id'] = 'wrongid'
|
43
|
+
bucket = get_bucket(config)
|
44
|
+
expect { described_class.bucket_valid?(bucket) }
|
45
|
+
.to raise_error(LogStash::ConfigurationError)
|
46
|
+
end
|
47
|
+
|
48
|
+
it 'tests with wrong key' do
|
49
|
+
config['secret_access_key'] = 'wrongaccesskey'
|
50
|
+
bucket = get_bucket(config)
|
51
|
+
expect { described_class.bucket_valid?(bucket) }
|
52
|
+
.to raise_error(LogStash::ConfigurationError)
|
53
|
+
end
|
54
|
+
|
55
|
+
it 'tests with wrong bucket name' do
|
56
|
+
config['bucket'] = 'wrongbucket'
|
57
|
+
bucket = get_bucket(config)
|
58
|
+
expect { described_class.bucket_valid?(bucket) }
|
59
|
+
.to raise_error(LogStash::ConfigurationError)
|
60
|
+
end
|
61
|
+
|
62
|
+
it 'tests with wrong reigon name' do
|
63
|
+
config['region'] = 'wrongregion'
|
64
|
+
bucket = get_bucket(config)
|
65
|
+
expect { described_class.bucket_valid?(bucket) }
|
66
|
+
.to raise_error(LogStash::ConfigurationError)
|
67
|
+
end
|
22
68
|
end
|
23
69
|
end
|
@@ -5,52 +5,10 @@ require 'logstash/outputs/qingstor/temporary_file'
|
|
5
5
|
require 'logstash/outputs/qingstor/rotation_policy'
|
6
6
|
|
7
7
|
describe LogStash::Outputs::Qingstor::RotationPolicy do
|
8
|
-
let(:size_file) {
|
8
|
+
let(:size_file) { 5 }
|
9
9
|
let(:time_file) { 2 }
|
10
|
-
let(:name) { 'foobar' }
|
11
|
-
let(:tmp_file) { Stud::Temporary.file }
|
12
|
-
let(:tmp_dir) { tmp_file.path }
|
13
|
-
let(:file) do
|
14
|
-
LogStash::Outputs::Qingstor::TemporaryFile.new(name, tmp_file, tmp_dir)
|
15
|
-
end
|
16
|
-
let(:content) { 'May the code be with you' * 100 }
|
17
|
-
|
18
|
-
context 'when size_and_time policy' do
|
19
|
-
subject { described_class.new('size_and_time', size_file, time_file) }
|
20
|
-
|
21
|
-
it 'raise error if time_file is no grater then 0' do
|
22
|
-
expect { described_class.new('size_and_time', 0, 0) }
|
23
|
-
.to raise_error(LogStash::ConfigurationError)
|
24
|
-
expect { described_class.new('size_and_time', -1, 0) }
|
25
|
-
.to raise_error(LogStash::ConfigurationError)
|
26
|
-
expect { described_class.new('size_and_time', 0, -1) }
|
27
|
-
.to raise_error(LogStash::ConfigurationError)
|
28
|
-
expect { described_class.new('size_and_time', -1, -1) }
|
29
|
-
.to raise_error(LogStash::ConfigurationError)
|
30
|
-
end
|
31
|
-
|
32
|
-
it 'return false if the file is not old enough' do
|
33
|
-
expect(subject.rotate?(file)).to be_falsey
|
34
|
-
end
|
35
|
-
|
36
|
-
it 'return false if the file is old enough with file size 0' do
|
37
|
-
allow(file)
|
38
|
-
.to receive(:ctime).and_return(Time.now - (time_file * 2 * 60))
|
39
|
-
expect(subject.rotate?(file)).to be_falsey
|
40
|
-
end
|
41
|
-
|
42
|
-
it 'return truth if the file is old enough and non-empty' do
|
43
|
-
file.write(content)
|
44
|
-
file.fsync
|
45
|
-
allow(file)
|
46
|
-
.to receive(:ctime).and_return(Time.now - (time_file * 2 * 60))
|
47
|
-
expect(subject.rotate?(file)).to be_truthy
|
48
|
-
end
|
49
|
-
end
|
50
|
-
|
51
|
-
context 'when size policy' do
|
52
|
-
subject { described_class.new('size', size_file, time_file) }
|
53
10
|
|
11
|
+
shared_examples 'size rotation' do
|
54
12
|
it 'raise error if size_file is no grater then 0' do
|
55
13
|
expect { described_class.new('size', 0, 0) }
|
56
14
|
.to raise_error(LogStash::ConfigurationError)
|
@@ -59,15 +17,23 @@ describe LogStash::Outputs::Qingstor::RotationPolicy do
|
|
59
17
|
end
|
60
18
|
|
61
19
|
it "return true if the file has a bigger size value then 'size_file'" do
|
62
|
-
file
|
63
|
-
file.
|
20
|
+
file = double('file')
|
21
|
+
allow(file).to receive(:ctime) { Time.now }
|
22
|
+
allow(file).to receive(:empty?) { true }
|
23
|
+
allow(file).to receive(:size) { 5 * 1024 * 1024 }
|
64
24
|
expect(subject.rotate?(file)).to be_truthy
|
65
25
|
end
|
66
|
-
end
|
67
26
|
|
68
|
-
|
69
|
-
|
27
|
+
it 'return false if the file size is zero' do
|
28
|
+
file = double('file')
|
29
|
+
allow(file).to receive(:ctime) { Time.now }
|
30
|
+
allow(file).to receive(:empty?) { true }
|
31
|
+
allow(file).to receive(:size) { 0 }
|
32
|
+
expect(subject.rotate?(file)).to be_falsey
|
33
|
+
end
|
34
|
+
end
|
70
35
|
|
36
|
+
shared_examples 'time rotation' do
|
71
37
|
it 'raise error if time_file is no grater then 0' do
|
72
38
|
expect { described_class.new('time', 0, 0) }
|
73
39
|
.to raise_error(LogStash::ConfigurationError)
|
@@ -76,21 +42,46 @@ describe LogStash::Outputs::Qingstor::RotationPolicy do
|
|
76
42
|
end
|
77
43
|
|
78
44
|
it 'return false if the file is not old enough' do
|
45
|
+
file = double('file')
|
46
|
+
allow(file).to receive(:ctime) { Time.now }
|
47
|
+
allow(file).to receive(:empty?) { false }
|
48
|
+
allow(file).to receive(:size) { 2 * 1024 * 1024 }
|
79
49
|
expect(subject.rotate?(file)).to be_falsey
|
80
50
|
end
|
81
51
|
|
82
52
|
it 'return false if the file is old enough with file size 0' do
|
83
|
-
|
84
|
-
|
53
|
+
file = double('file')
|
54
|
+
allow(file).to receive(:ctime) { Time.now - (time_file * 2 * 60) }
|
55
|
+
allow(file).to receive(:empty?) { true }
|
56
|
+
allow(file).to receive(:size) { 0 }
|
85
57
|
expect(subject.rotate?(file)).to be_falsey
|
86
58
|
end
|
87
59
|
|
88
60
|
it 'return truth if the file is old enough and non-empty' do
|
89
|
-
file
|
90
|
-
file.
|
91
|
-
allow(file)
|
92
|
-
|
61
|
+
file = double('file')
|
62
|
+
allow(file).to receive(:ctime) { Time.now - (time_file * 2 * 60) }
|
63
|
+
allow(file).to receive(:empty?) { false }
|
64
|
+
allow(file).to receive(:size) { 5 * 1024 * 1024 }
|
93
65
|
expect(subject.rotate?(file)).to be_truthy
|
94
66
|
end
|
95
67
|
end
|
68
|
+
|
69
|
+
context 'when time policy' do
|
70
|
+
subject { described_class.new('time', size_file, time_file) }
|
71
|
+
|
72
|
+
include_examples 'time rotation'
|
73
|
+
end
|
74
|
+
|
75
|
+
context 'when size policy' do
|
76
|
+
subject { described_class.new('size', size_file, time_file) }
|
77
|
+
|
78
|
+
include_examples 'size rotation'
|
79
|
+
end
|
80
|
+
|
81
|
+
context 'when size_and_time policy' do
|
82
|
+
subject { described_class.new('size_and_time', size_file, time_file) }
|
83
|
+
|
84
|
+
include_examples 'time rotation'
|
85
|
+
include_examples 'size rotation'
|
86
|
+
end
|
96
87
|
end
|
@@ -7,11 +7,18 @@ require 'fileutils'
|
|
7
7
|
require 'tmpdir'
|
8
8
|
|
9
9
|
describe LogStash::Outputs::Qingstor::TemporaryFileFactory do
|
10
|
-
|
10
|
+
def tmp_dir
|
11
|
+
File.join(Dir.tmpdir, 'logstash-qs')
|
12
|
+
end
|
13
|
+
|
14
|
+
subject { described_class.new(prefix, tags, encoding, tmp_dir) }
|
11
15
|
|
12
16
|
let(:prefix) { 'lg2qs' }
|
13
17
|
let(:tags) { [] }
|
14
|
-
|
18
|
+
|
19
|
+
after(:all) do
|
20
|
+
FileUtils.rm_rf(tmp_dir)
|
21
|
+
end
|
15
22
|
|
16
23
|
shared_examples 'file factory' do
|
17
24
|
it 'creates the file on disk' do
|
@@ -27,7 +27,7 @@ describe LogStash::Outputs::Qingstor do
|
|
27
27
|
'prefix' => prefix
|
28
28
|
}
|
29
29
|
end
|
30
|
-
|
30
|
+
|
31
31
|
before(:all) do
|
32
32
|
FileUtils.mkdir_p(tmp_dir_path) unless File.exist?(tmp_dir_path)
|
33
33
|
end
|
@@ -45,7 +45,7 @@ describe LogStash::Outputs::Qingstor do
|
|
45
45
|
expect(list_remote_file.size).to eq(1)
|
46
46
|
end
|
47
47
|
|
48
|
-
it 'use aes256 to encrpytion in the server side'
|
48
|
+
it 'use aes256 to encrpytion in the server side' do
|
49
49
|
cipher = OpenSSL::Cipher::AES256.new(:CBC)
|
50
50
|
cipher.encrypt
|
51
51
|
key = cipher.random_key
|
@@ -56,7 +56,6 @@ describe LogStash::Outputs::Qingstor do
|
|
56
56
|
end
|
57
57
|
|
58
58
|
it 'upload existing file if turn on restore function' do
|
59
|
-
binding.pry
|
60
59
|
non_empty_file = File.open(File.join(tmpdir, 'non-empty-file'), 'w')
|
61
60
|
non_empty_file.write(event_encoded * 10)
|
62
61
|
non_empty_file.close
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-output-qingstor
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Evan Zhao
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-01-
|
11
|
+
date: 2018-01-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
@@ -41,6 +41,9 @@ dependencies:
|
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
requirement: !ruby/object:Gem::Requirement
|
43
43
|
requirements:
|
44
|
+
- - "~>"
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '1.9'
|
44
47
|
- - ">="
|
45
48
|
- !ruby/object:Gem::Version
|
46
49
|
version: 1.9.3
|
@@ -49,6 +52,9 @@ dependencies:
|
|
49
52
|
type: :runtime
|
50
53
|
version_requirements: !ruby/object:Gem::Requirement
|
51
54
|
requirements:
|
55
|
+
- - "~>"
|
56
|
+
- !ruby/object:Gem::Version
|
57
|
+
version: '1.9'
|
52
58
|
- - ">="
|
53
59
|
- !ruby/object:Gem::Version
|
54
60
|
version: 1.9.3
|
@@ -129,7 +135,7 @@ files:
|
|
129
135
|
- spec/outputs/spec_helper.rb
|
130
136
|
homepage: https://github.com/yunify/logstash-output-qingstor
|
131
137
|
licenses:
|
132
|
-
- Apache
|
138
|
+
- Apache-2.0
|
133
139
|
metadata:
|
134
140
|
logstash_plugin: 'true'
|
135
141
|
logstash_group: output
|