fluent-plugin-insert-id 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +7 -0
- data/README.md +55 -7
- data/fluent-plugin-insert-id.gemspec +5 -6
- data/lib/fluent/plugin/filter_insert_id.rb +2 -2
- metadata +8 -22
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3f8c9bed82fc7deb1ce5d231f9ec119ab6c581202e1b1549e940528de458a4f3
|
4
|
+
data.tar.gz: 4c4ce9dee089745f4ab6e8b1f07634fa3fd388c6073c6dfcf077608b960df65a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8ef0f27050c7ec286b60adc83fc82539d6d33375c86a2a26ee9b0e3e55b879084bf8293205d7f3eb710ffb7caaf5e9cf4ebb6bfa7b1bedbcbe07880f025cdffe
|
7
|
+
data.tar.gz: 242363d6a2af8ac79b2b5ca2212ec97d02e01a489f13ce8c5d8a7c2982f0d43a58fee6ba2643778fd56cbcbc793738b803823bf994f33d38070d313ca643173f
|
data/.gitignore
CHANGED
data/README.md
CHANGED
@@ -1,9 +1,10 @@
|
|
1
1
|
# fluent-plugin-insert-id
|
2
2
|
|
3
|
-
[Fluentd](https://fluentd.org/) filter plugin to insert unique ID string.
|
3
|
+
[Fluentd](https://fluentd.org/) filter plugin to insert unique ID string into the message.
|
4
4
|
|
5
5
|
The original implementation was copied from [fluent-plugin-google-cloud](https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud)
|
6
|
-
which was created by Google to handle GKE's official Stackdriver Logging reporting in sidecar
|
6
|
+
which was created by Google to handle GCP GKE's official Stackdriver Logging reporting in sidecar container for each pod.
|
7
|
+
Original source requires fluentd v0.12 and doesn't support fluentd v1.0 but this plugin only supports fluentd v1.0.
|
7
8
|
|
8
9
|
## How it works
|
9
10
|
```
|
@@ -22,7 +23,7 @@ which was created by Google to handle GKE's official Stackdriver Logging reporti
|
|
22
23
|
2019-08-25 21:20:50.035415329 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetdea"}
|
23
24
|
```
|
24
25
|
|
25
|
-
*
|
26
|
+
* Inserted ID contains 0-9 and a-z lowercase characters.
|
26
27
|
* Initial ID generated is random string like "nu8a3ptahpbetddc".
|
27
28
|
* Series ID after the initial ID are 'incremented' string which uses Ruby's `String.next()`.
|
28
29
|
* 'incremented' string also has 'carry' feature. Please check below links for more details.
|
@@ -30,6 +31,40 @@ which was created by Google to handle GKE's official Stackdriver Logging reporti
|
|
30
31
|
* [String.next() (Japanese)](https://docs.ruby-lang.org/ja/2.4.0/class/String.html#I_NEXT)
|
31
32
|
* This ordered ID makes debugging easier in most cases.
|
32
33
|
|
34
|
+
#### ID string length
|
35
|
+
* **From version 1.1.0, it is guaranteed that ID is fixed length string. **
|
36
|
+
* **In version 1.0.0, the ID string length is incremented when carry happens at left-most characters.**
|
37
|
+
|
38
|
+
```ruby
|
39
|
+
# Version 1.0.0
|
40
|
+
{"a":"foo","b":"bar","insert-id":"z99999999999999z"}
|
41
|
+
{"a":"foo","b":"bar","insert-id":"aa00000000000000z"} # Left most character carry adds new digit.
|
42
|
+
|
43
|
+
# Version 1.1.0
|
44
|
+
{"a":"foo","b":"bar","insert-id":"z99999999999999z"}
|
45
|
+
{"a":"foo","b":"bar","insert-id":"a00000000000000z"} # Left most character carry is ignored.
|
46
|
+
```
|
47
|
+
|
48
|
+
#### Existing ID protection
|
49
|
+
If the message already has the key for inserted ID, the filter doesn't touch it and
|
50
|
+
existing value is protected.
|
51
|
+
|
52
|
+
```
|
53
|
+
2019-08-27 02:10:07.422911774 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsv"}
|
54
|
+
2019-08-27 02:10:08.129842499 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsw"}
|
55
|
+
2019-08-27 02:10:08.940316454 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsx"}
|
56
|
+
2019-08-27 02:11:02.498772740 +0000 message.test: {"a":"foo","b":"bar","insert-id":"existing_ID"}
|
57
|
+
2019-08-27 02:11:06.802934944 +0000 message.test: {"a":"foo","b":"bar","insert-id":"ehrbwzp772xitjsy"}
|
58
|
+
```
|
59
|
+
|
60
|
+
## Requirements
|
61
|
+
|
62
|
+
| fluent-plugin-insert-id | fluentd | ruby |
|
63
|
+
|--------------------------|---------|------|
|
64
|
+
| >= 1.0.0 | >= v0.14.x | >= 2.1 |
|
65
|
+
|
66
|
+
fluentd v0.12 is not supported.
|
67
|
+
|
33
68
|
## Installation
|
34
69
|
|
35
70
|
### RubyGems
|
@@ -54,13 +89,26 @@ $ bundle
|
|
54
89
|
|
55
90
|
## Configuration
|
56
91
|
|
57
|
-
|
92
|
+
**insert_id_key** (string) (optional)
|
58
93
|
|
59
|
-
|
60
|
-
|
94
|
+
The key of inserted-id.
|
95
|
+
Default value is `"insert-id"`.
|
96
|
+
|
97
|
+
## Usage : Insert ID with default key
|
98
|
+
```aconf
|
99
|
+
<filter **>
|
100
|
+
@type insert_id
|
101
|
+
</filter>
|
61
102
|
```
|
62
103
|
|
63
|
-
|
104
|
+
## Usage : Insert ID with custom key
|
105
|
+
In this case, key of ID is changed to 'message_id'
|
106
|
+
```aconf
|
107
|
+
<filter **>
|
108
|
+
@type insert_id
|
109
|
+
insert_id_key message_id
|
110
|
+
</filter>
|
111
|
+
```
|
64
112
|
|
65
113
|
## Copyright
|
66
114
|
|
@@ -3,12 +3,12 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
3
3
|
|
4
4
|
Gem::Specification.new do |spec|
|
5
5
|
spec.name = "fluent-plugin-insert-id"
|
6
|
-
spec.version = "1.
|
6
|
+
spec.version = "1.1.0"
|
7
7
|
spec.authors = ["Kamome Shido"]
|
8
8
|
spec.email = ["shidokamo@users.noreply.github.com"]
|
9
9
|
|
10
|
-
spec.summary = %q{fluentd filter plugin to insert unique id}
|
11
|
-
spec.description = %q{fluentd filter plugin to insert unique id}
|
10
|
+
spec.summary = %q{fluentd filter plugin to insert unique id into the message}
|
11
|
+
spec.description = %q{fluentd filter plugin to insert unique id into the message}
|
12
12
|
spec.homepage = "https://github.com/shidokamo/fluent-plugin-insert-id"
|
13
13
|
spec.license = "Apache-2.0"
|
14
14
|
|
@@ -20,8 +20,7 @@ Gem::Specification.new do |spec|
|
|
20
20
|
spec.test_files = test_files
|
21
21
|
spec.require_paths = ["lib"]
|
22
22
|
|
23
|
-
spec.add_development_dependency "
|
24
|
-
spec.add_development_dependency "
|
25
|
-
spec.add_development_dependency "test-unit", "~> 3.0"
|
23
|
+
spec.add_development_dependency "rake", ">= 12.0"
|
24
|
+
spec.add_development_dependency "test-unit", ">= 3.0"
|
26
25
|
spec.add_runtime_dependency "fluentd", [">= 0.14.10", "< 2"]
|
27
26
|
end
|
@@ -57,7 +57,7 @@ module Fluent
|
|
57
57
|
def start
|
58
58
|
super
|
59
59
|
# Initialize the ID
|
60
|
-
log.info "Started the
|
60
|
+
log.info "Started the insert-id plugin with #{@insert_id_key} as the insert ID key."
|
61
61
|
@insert_id = generate_initial_insert_id
|
62
62
|
log.info "Initialized the insert ID key to #{@insert_id}."
|
63
63
|
end
|
@@ -85,7 +85,7 @@ module Fluent
|
|
85
85
|
|
86
86
|
# Increment the insertId and return the new value.
|
87
87
|
def increment_insert_id
|
88
|
-
@insert_id = @insert_id.next
|
88
|
+
@insert_id = @insert_id.next.slice(-INSERT_ID_SIZE,INSERT_ID_SIZE) # slice is required to make sure ID has fixed length.
|
89
89
|
end
|
90
90
|
end
|
91
91
|
end
|
metadata
CHANGED
@@ -1,55 +1,41 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-insert-id
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Kamome Shido
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2019-08-
|
11
|
+
date: 2019-08-27 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
|
-
- !ruby/object:Gem::Dependency
|
14
|
-
name: bundler
|
15
|
-
requirement: !ruby/object:Gem::Requirement
|
16
|
-
requirements:
|
17
|
-
- - "~>"
|
18
|
-
- !ruby/object:Gem::Version
|
19
|
-
version: '1.14'
|
20
|
-
type: :development
|
21
|
-
prerelease: false
|
22
|
-
version_requirements: !ruby/object:Gem::Requirement
|
23
|
-
requirements:
|
24
|
-
- - "~>"
|
25
|
-
- !ruby/object:Gem::Version
|
26
|
-
version: '1.14'
|
27
13
|
- !ruby/object:Gem::Dependency
|
28
14
|
name: rake
|
29
15
|
requirement: !ruby/object:Gem::Requirement
|
30
16
|
requirements:
|
31
|
-
- - "
|
17
|
+
- - ">="
|
32
18
|
- !ruby/object:Gem::Version
|
33
19
|
version: '12.0'
|
34
20
|
type: :development
|
35
21
|
prerelease: false
|
36
22
|
version_requirements: !ruby/object:Gem::Requirement
|
37
23
|
requirements:
|
38
|
-
- - "
|
24
|
+
- - ">="
|
39
25
|
- !ruby/object:Gem::Version
|
40
26
|
version: '12.0'
|
41
27
|
- !ruby/object:Gem::Dependency
|
42
28
|
name: test-unit
|
43
29
|
requirement: !ruby/object:Gem::Requirement
|
44
30
|
requirements:
|
45
|
-
- - "
|
31
|
+
- - ">="
|
46
32
|
- !ruby/object:Gem::Version
|
47
33
|
version: '3.0'
|
48
34
|
type: :development
|
49
35
|
prerelease: false
|
50
36
|
version_requirements: !ruby/object:Gem::Requirement
|
51
37
|
requirements:
|
52
|
-
- - "
|
38
|
+
- - ">="
|
53
39
|
- !ruby/object:Gem::Version
|
54
40
|
version: '3.0'
|
55
41
|
- !ruby/object:Gem::Dependency
|
@@ -72,7 +58,7 @@ dependencies:
|
|
72
58
|
- - "<"
|
73
59
|
- !ruby/object:Gem::Version
|
74
60
|
version: '2'
|
75
|
-
description: fluentd filter plugin to insert unique id
|
61
|
+
description: fluentd filter plugin to insert unique id into the message
|
76
62
|
email:
|
77
63
|
- shidokamo@users.noreply.github.com
|
78
64
|
executables: []
|
@@ -110,7 +96,7 @@ requirements: []
|
|
110
96
|
rubygems_version: 3.0.3
|
111
97
|
signing_key:
|
112
98
|
specification_version: 4
|
113
|
-
summary: fluentd filter plugin to insert unique id
|
99
|
+
summary: fluentd filter plugin to insert unique id into the message
|
114
100
|
test_files:
|
115
101
|
- test/helper.rb
|
116
102
|
- test/plugin/test_filter_insert_id.rb
|