logstash-filter-fingerprint 3.1.1 → 3.2.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 38afa5034d1df32f7d6b1b15208202c04d4a1de7
4
- data.tar.gz: bdceb08719fd846cbfe6b82bfab4a57ba858ae6c
2
+ SHA256:
3
+ metadata.gz: 5985afeb097946794b24fa48da9e78f4a4146e39b7a39b4be98f11c2b74cd7f8
4
+ data.tar.gz: 1e6ba943dbae9448dacf4bcf8f4d268aa48381d816be73630f6594358090c8d4
5
5
  SHA512:
6
- metadata.gz: 75e44348b0abc582ffad874105a5fe0f0467a1ee60047e2c03d22b4955f390cfd0bbc76a87b9541a928633da2535e25f1ae3a1053006b0bf1f82aa83921fed8c
7
- data.tar.gz: 16ab2a6e9e604e441c811054a98c1ea0a803232edef589d0ee9f856fb7b8c421c57331116c0dc53d84b648c2f4c36178b75df649d7b94cafcfe0e905586a6512
6
+ metadata.gz: 63d38e83879e80b301af57fd82fbf60630ace866b6f102425e8d3ae97467b5384b17268215e52854c35bf1c24e7dfe8f6bd448fa9821f7289610c794232d7a48
7
+ data.tar.gz: 566ef91a8e6d67abf113bdd07621f65d47a290d0b12f7278d881d3eb13cdaf8c6346147eee3d4534a4184ec70544440177b9821b54e61f58ace71a84ae821669
data/CHANGELOG.md CHANGED
@@ -1,3 +1,18 @@
1
+ ## 3.2.3
2
+ - [DOC] Expanded description for concatenate_sources behavior and provided examples [#60](https://github.com/logstash-plugins/logstash-filter-fingerprint/pull/60)
3
+
4
+ ## 3.2.2
5
+ - Fixed lack of consistent fingerprints on Hash/Map objects [#55](https://github.com/logstash-plugins/logstash-filter-fingerprint/pull/55)
6
+
7
+ ## 3.2.1
8
+ - Fixed concurrent SHA fingerprinting by making the instances thread local
9
+
10
+ ## 3.2.0
11
+ - Added support for non-keyed, regular hash functions [#18](https://github.com/logstash-plugins/logstash-filter-fingerprint/issues/18)
12
+
13
+ ## 3.1.2
14
+ - Update gemspec summary
15
+
1
16
  ## 3.1.1
2
17
  - Fix some documentation issues
3
18
 
data/CONTRIBUTORS CHANGED
@@ -8,6 +8,8 @@ Contributors:
8
8
  * Richard Pijnenburg (electrical)
9
9
  * Suyog Rao (suyograo)
10
10
  * Tray (torrancew)
11
+ * praseodym
12
+
11
13
 
12
14
  Note: If you've sent us patches, bug reports, or otherwise contributed to
13
15
  Logstash, and you aren't on the list above and want to be, please let us know
data/LICENSE CHANGED
@@ -1,13 +1,202 @@
1
- Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
2
1
 
3
- Licensed under the Apache License, Version 2.0 (the "License");
4
- you may not use this file except in compliance with the License.
5
- You may obtain a copy of the License at
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
6
5
 
7
- http://www.apache.org/licenses/LICENSE-2.0
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
8
7
 
9
- Unless required by applicable law or agreed to in writing, software
10
- distributed under the License is distributed on an "AS IS" BASIS,
11
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
- See the License for the specific language governing permissions and
13
- limitations under the License.
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright 2020 Elastic and contributors
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Logstash Plugin
2
2
 
3
- [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-filter-fingerprint.svg)](https://travis-ci.org/logstash-plugins/logstash-filter-fingerprint)
3
+ [![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-filter-fingerprint.svg)](https://travis-ci.com/logstash-plugins/logstash-filter-fingerprint)
4
4
 
5
5
  This is a plugin for [Logstash](https://github.com/elastic/logstash).
6
6
 
data/docs/index.asciidoc CHANGED
@@ -23,17 +23,14 @@ include::{include_path}/plugin_header.asciidoc[]
23
23
  Create consistent hashes (fingerprints) of one or more fields and store
24
24
  the result in a new field.
25
25
 
26
- This can e.g. be used to create consistent document ids when inserting
27
- events into Elasticsearch, allowing events in Logstash to cause existing
28
- documents to be updated rather than new documents to be created.
29
-
30
- NOTE: When using any method other than 'UUID', 'PUNCTUATION' or 'MURMUR3'
31
- you must set the key, otherwise the plugin will raise an exception
26
+ You can use this plugin to create consistent document ids when events are
27
+ inserted into Elasticsearch. This approach means that existing documents can be
28
+ updated instead of creating new documents.
32
29
 
33
30
  NOTE: When the `target` option is set to `UUID` the result won't be
34
31
  a consistent hash but a random
35
32
  https://en.wikipedia.org/wiki/Universally_unique_identifier[UUID].
36
- To generate UUIDs, prefer the <<plugins-filters-uuid,uuid filter>>.
33
+ To generate UUIDs, prefer the {logstash-ref}/plugins-filters-uuid.html[uuid filter].
37
34
 
38
35
  [id="plugins-{type}s-{plugin}-options"]
39
36
  ==== Fingerprint Filter Configuration Options
@@ -75,12 +72,56 @@ base64 encoded rather than hex encoded strings.
75
72
  When set to `true` and `method` isn't `UUID` or `PUNCTUATION`, the
76
73
  plugin concatenates the names and values of all fields given in the
77
74
  `source` option into one string (like the old checksum filter) before
78
- doing the fingerprint computation. If `false` and multiple source
79
- fields are given, the target field will be an array with fingerprints
80
- of the source fields given.
75
+ doing the fingerprint computation.
76
+
77
+ If `false` and multiple source fields are given, the target field will be single
78
+ fingerprint of the last source field.
79
+
80
+ **Example: `concatenate_sources`=false**
81
+
82
+ This example produces a single fingerprint that is computed from "birthday," the
83
+ last source field.
84
+
85
+ [source,ruby]
86
+ -----
87
+ fingerprint {
88
+ source => ["user_id", "siblings", "birthday"]
89
+ }
90
+ -----
91
+
92
+ The output is:
93
+
94
+ [source,ruby]
95
+ -----
96
+ "fingerprint" => "6b6390a4416131f82b6ffb509f6e779e5dd9630f".
97
+ -----
98
+
99
+ **Example: `concatenate_sources`=false with array**
100
+
101
+ If the last source field is an array, you get an array of fingerprints.
102
+
103
+ In this example, "siblings" is an array ["big brother", "little sister", "little brother"].
104
+
105
+ [source,ruby]
106
+ -----
107
+ fingerprint {
108
+ source => ["user_id", "siblings"]
109
+ }
110
+ -----
111
+
112
+ The output is:
113
+
114
+ [source,ruby]
115
+ -----
116
+ "fingerprint" => [
117
+ [0] "8a8a9323677f4095fcf0c8c30b091a0133b00641",
118
+ [1] "2ce11b313402e0e9884e094409f8d9fcf01337c2",
119
+ [2] "adc0b90f9391a82098c7b99e66a816e9619ad0a7"
120
+ ],
121
+ -----
81
122
 
82
123
  [id="plugins-{type}s-{plugin}-concatenate_all_fields"]
83
- ===== `concatenate_sources`
124
+ ===== `concatenate_all_fields`
84
125
 
85
126
  * Value type is <<boolean,boolean>>
86
127
  * Default value is `false`
@@ -99,8 +140,7 @@ source fields given.
99
140
  * There is no default value for this setting.
100
141
 
101
142
  When used with the `IPV4_NETWORK` method fill in the subnet prefix length.
102
- Key is required with all methods except `MURMUR3`, `PUNCTUATION` or `UUID`.
103
- With other methods fill in the HMAC key.
143
+ With other methods, optionally fill in the HMAC key.
104
144
 
105
145
  [id="plugins-{type}s-{plugin}-method"]
106
146
  ===== `method`
@@ -111,10 +151,12 @@ With other methods fill in the HMAC key.
111
151
 
112
152
  The fingerprint method to use.
113
153
 
114
- If set to `SHA1`, `SHA256`, `SHA384`, `SHA512`, or `MD5` the
115
- cryptographic keyed-hash function with the same name will be used to
116
- generate the fingerprint. If set to `MURMUR3` the non-cryptographic
117
- MurmurHash function will be used.
154
+ If set to `SHA1`, `SHA256`, `SHA384`, `SHA512`, or `MD5` and a key is set,
155
+ the cryptographic hash function with the same name will be used to generate
156
+ the fingerprint. When a key set, the keyed-hash (HMAC) digest function will
157
+ be used.
158
+
159
+ If set to `MURMUR3` the non-cryptographic MurmurHash function will be used.
118
160
 
119
161
  If set to `IPV4_NETWORK` the input data needs to be a IPv4 address and
120
162
  the hash value will be the masked-out address using the number of bits
@@ -150,4 +192,4 @@ Any current contents of that field will be overwritten.
150
192
 
151
193
 
152
194
  [id="plugins-{type}s-{plugin}-common-options"]
153
- include::{include_path}/{type}.asciidoc[]
195
+ include::{include_path}/{type}.asciidoc[]
@@ -34,8 +34,7 @@ class LogStash::Filters::Fingerprint < LogStash::Filters::Base
34
34
  config :target, :validate => :string, :default => 'fingerprint'
35
35
 
36
36
  # When used with the `IPV4_NETWORK` method fill in the subnet prefix length.
37
- # Key is required with all methods except `MURMUR3`, `PUNCTUATION` or `UUID`.
38
- # With other methods fill in the HMAC key.
37
+ # With other methods, optionally fill in the HMAC key.
39
38
  config :key, :validate => :string
40
39
 
41
40
  # When set to `true`, the `SHA1`, `SHA256`, `SHA384`, `SHA512` and `MD5` fingerprint methods will produce
@@ -44,10 +43,12 @@ class LogStash::Filters::Fingerprint < LogStash::Filters::Base
44
43
 
45
44
  # The fingerprint method to use.
46
45
  #
47
- # If set to `SHA1`, `SHA256`, `SHA384`, `SHA512`, or `MD5` the
48
- # cryptographic keyed-hash function with the same name will be used to
49
- # generate the fingerprint. If set to `MURMUR3` the non-cryptographic
50
- # MurmurHash function will be used.
46
+ # If set to `SHA1`, `SHA256`, `SHA384`, `SHA512`, or `MD5` and a key is set,
47
+ # the cryptographic hash function with the same name will be used to generate
48
+ # the fingerprint. When a key set, the keyed-hash (HMAC) digest function will
49
+ # be used.
50
+ #
51
+ # If set to `MURMUR3` the non-cryptographic MurmurHash function will be used.
51
52
  #
52
53
  # If set to `IPV4_NETWORK` the input data needs to be a IPv4 address and
53
54
  # the hash value will be the masked-out address using the number of bits
@@ -79,7 +80,7 @@ class LogStash::Filters::Fingerprint < LogStash::Filters::Base
79
80
  # convert to symbol for faster comparisons
80
81
  @method = @method.to_sym
81
82
 
82
- # require any library and set the anonymize function
83
+ # require any library and set the fingerprint function
83
84
  case @method
84
85
  when :IPV4_NETWORK
85
86
  if @key.nil?
@@ -90,24 +91,15 @@ class LogStash::Filters::Fingerprint < LogStash::Filters::Base
90
91
  :error => "Key value is empty. please fill in a subnet prefix length"
91
92
  )
92
93
  end
93
- class << self; alias_method :anonymize, :anonymize_ipv4_network; end
94
+ class << self; alias_method :fingerprint, :fingerprint_ipv4_network; end
94
95
  when :MURMUR3
95
- class << self; alias_method :anonymize, :anonymize_murmur3; end
96
+ class << self; alias_method :fingerprint, :fingerprint_murmur3; end
96
97
  when :UUID
97
98
  # nothing
98
99
  when :PUNCTUATION
99
100
  # nothing
100
101
  else
101
- if @key.nil?
102
- raise LogStash::ConfigurationError, I18n.t(
103
- "logstash.runner.configuration.invalid_plugin_register",
104
- :plugin => "filter",
105
- :type => "fingerprint",
106
- :error => "Key value is empty. Please fill in an encryption key"
107
- )
108
- end
109
- class << self; alias_method :anonymize, :anonymize_openssl; end
110
- @digest = select_digest(@method)
102
+ class << self; alias_method :fingerprint, :fingerprint_openssl; end
111
103
  end
112
104
  end
113
105
 
@@ -127,24 +119,24 @@ class LogStash::Filters::Fingerprint < LogStash::Filters::Base
127
119
  if @concatenate_sources || @concatenate_all_fields
128
120
  to_string = ""
129
121
  if @concatenate_all_fields
130
- event.to_hash.sort.map do |k,v|
122
+ deep_sort_hashes(event.to_hash).each do |k,v|
131
123
  to_string << "|#{k}|#{v}"
132
124
  end
133
125
  else
134
126
  @source.sort.each do |k|
135
- to_string << "|#{k}|#{event.get(k)}"
127
+ to_string << "|#{k}|#{deep_sort_hashes(event.get(k))}"
136
128
  end
137
129
  end
138
130
  to_string << "|"
139
131
  @logger.debug? && @logger.debug("String built", :to_checksum => to_string)
140
- event.set(@target, anonymize(to_string))
132
+ event.set(@target, fingerprint(to_string))
141
133
  else
142
134
  @source.each do |field|
143
135
  next unless event.include?(field)
144
136
  if event.get(field).is_a?(Array)
145
- event.set(@target, event.get(field).collect { |v| anonymize(v) })
137
+ event.set(@target, event.get(field).collect { |v| fingerprint(deep_sort_hashes(v)) })
146
138
  else
147
- event.set(@target, anonymize(event.get(field)))
139
+ event.set(@target, fingerprint(deep_sort_hashes(event.get(field))))
148
140
  end
149
141
  end
150
142
  end
@@ -153,23 +145,53 @@ class LogStash::Filters::Fingerprint < LogStash::Filters::Base
153
145
  end
154
146
 
155
147
  private
148
+ def deep_sort_hashes(object)
149
+ case object
150
+ when Hash
151
+ sorted_hash = Hash.new
152
+ object.sort.each do |sorted_key, value|
153
+ sorted_hash[sorted_key] = deep_sort_hashes(value)
154
+ end
155
+ sorted_hash
156
+ when Array
157
+ object.map {|element| deep_sort_hashes(element) }
158
+ else
159
+ object
160
+ end
161
+ end
156
162
 
157
- def anonymize_ipv4_network(ip_string)
163
+ def fingerprint_ipv4_network(ip_string)
158
164
  # in JRuby 1.7.11 outputs as US-ASCII
159
165
  IPAddr.new(ip_string).mask(@key.to_i).to_s.force_encoding(Encoding::UTF_8)
160
166
  end
161
167
 
162
- def anonymize_openssl(data)
168
+ def fingerprint_openssl(data)
169
+ # since OpenSSL::Digest instances aren't thread safe, we must ensure that
170
+ # each pipeline worker thread gets its own instance.
171
+ # Also, since a logstash pipeline may contain multiple fingerprint filters
172
+ # we must include the id in the thread local variable name, so that we can
173
+ # store multiple digest instances
174
+ digest_string = "digest-#{id}"
175
+ Thread.current[digest_string] ||= select_digest(@method)
176
+ digest = Thread.current[digest_string]
163
177
  # in JRuby 1.7.11 outputs as ASCII-8BIT
164
- if @base64encode
165
- hash = OpenSSL::HMAC.digest(@digest, @key, data.to_s)
166
- Base64.strict_encode64(hash).force_encoding(Encoding::UTF_8)
178
+ if @key.nil?
179
+ if @base64encode
180
+ digest.base64digest(data.to_s).force_encoding(Encoding::UTF_8)
181
+ else
182
+ digest.hexdigest(data.to_s).force_encoding(Encoding::UTF_8)
183
+ end
167
184
  else
168
- OpenSSL::HMAC.hexdigest(@digest, @key, data.to_s).force_encoding(Encoding::UTF_8)
185
+ if @base64encode
186
+ hash = OpenSSL::HMAC.digest(digest, @key, data.to_s)
187
+ Base64.strict_encode64(hash).force_encoding(Encoding::UTF_8)
188
+ else
189
+ OpenSSL::HMAC.hexdigest(digest, @key, data.to_s).force_encoding(Encoding::UTF_8)
190
+ end
169
191
  end
170
192
  end
171
193
 
172
- def anonymize_murmur3(value)
194
+ def fingerprint_murmur3(value)
173
195
  case value
174
196
  when Fixnum
175
197
  MurmurHash3::V32.int_hash(value)
@@ -1,9 +1,9 @@
1
1
  Gem::Specification.new do |s|
2
2
 
3
3
  s.name = 'logstash-filter-fingerprint'
4
- s.version = '3.1.1'
5
- s.licenses = ['Apache License (2.0)']
6
- s.summary = "Fingerprint fields using by replacing values with a consistent hash."
4
+ s.version = '3.2.3'
5
+ s.licenses = ['Apache-2.0']
6
+ s.summary = "Fingerprints fields by replacing values with a consistent hash"
7
7
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
8
8
  s.authors = ["Elastic"]
9
9
  s.email = 'info@elastic.co'
@@ -4,321 +4,284 @@ require "logstash/filters/fingerprint"
4
4
 
5
5
  describe LogStash::Filters::Fingerprint do
6
6
 
7
- describe "fingerprint ipaddress with IPV4_NETWORK method" do
8
- config <<-CONFIG
9
- filter {
10
- fingerprint {
11
- source => ["clientip"]
12
- method => "IPV4_NETWORK"
13
- key => 24
14
- }
15
- }
16
- CONFIG
17
-
18
- sample("clientip" => "233.255.13.44") do
19
- insist { subject.get("fingerprint") } == "233.255.13.0"
20
- end
7
+ let(:plugin) { described_class.new(config) }
8
+ let(:config) { { "method" => fingerprint_method } }
9
+ let(:fingerprint_method) { "SHA1" } # default
10
+ let(:data) { {} }
11
+ let(:event) { LogStash::Event.new(data) }
12
+ let(:fingerprint) { event.get("fingerprint") }
13
+
14
+ before(:each) do
15
+ plugin.register
16
+ plugin.filter(event)
21
17
  end
22
18
 
23
- describe "fingerprint string with MURMUR3 method" do
24
- config <<-CONFIG
25
- filter {
26
- fingerprint {
27
- source => ["clientip"]
28
- method => "MURMUR3"
29
- }
30
- }
31
- CONFIG
19
+ context "with a string field" do
20
+ let(:data) { {"clientip" => "123.123.123.123" } }
21
+ let(:config) { super().merge("source" => ["clientip" ]) }
32
22
 
33
- sample("clientip" => "123.52.122.33") do
34
- insist { subject.get("fingerprint") } == 1541804874
35
- end
36
- end
37
-
38
- describe "fingerprint string with SHA1 alogrithm" do
39
- config <<-CONFIG
40
- filter {
41
- fingerprint {
42
- source => ["clientip"]
43
- key => "longencryptionkey"
44
- method => 'SHA1'
45
- }
46
- }
47
- CONFIG
23
+ describe "the IPV4_NETWORK method" do
24
+ let(:fingerprint_method) { "IPV4_NETWORK" }
25
+ let(:config) { super().merge("key" => 24) }
48
26
 
49
- sample("clientip" => "123.123.123.123") do
50
- insist { subject.get("fingerprint") } == "fdc60acc4773dc5ac569ffb78fcb93c9630797f4"
27
+ it "fingerprints the ip as the network" do
28
+ expect(fingerprint).to eq("123.123.123.0")
29
+ end
51
30
  end
52
- end
53
31
 
54
- describe "fingerprint string with SHA1 alogrithm on all event fields" do
55
- config <<-CONFIG
56
- filter {
57
- fingerprint {
58
- concatenate_all_fields => true
59
- key => "longencryptionkey"
60
- method => 'SHA1'
61
- }
62
- }
63
- CONFIG
32
+ describe "the MURMUR3 method" do
33
+ let(:fingerprint_method) { "MURMUR3" }
64
34
 
65
- # The @timestamp field is specified in this sample event as we need the event contents to be constant for the tests
66
- sample("@timestamp" => "2017-07-26T14:44:27.064Z", "clientip" => "123.123.123.123", "message" => "This is a test message", "log_level" => "INFO", "offset" => 123456789, "type" => "test") do
67
- insist { subject.get("fingerprint") } == "d7c617f4d40b2cb677a7003af68a41c415f58031"
35
+ it "fingerprints the value" do
36
+ expect(fingerprint).to eq(4013733395)
37
+ end
68
38
  end
69
- end
70
39
 
71
- describe "fingerprint string with SHA1 alogrithm and base64 encoding" do
72
- config <<-CONFIG
73
- filter {
74
- fingerprint {
75
- source => ["clientip"]
76
- key => "longencryptionkey"
77
- method => 'SHA1'
78
- base64encode => true
79
- }
80
- }
81
- CONFIG
82
-
83
- sample("clientip" => "123.123.123.123") do
84
- insist { subject.get("fingerprint") } == "/cYKzEdz3FrFaf+3j8uTyWMHl/Q="
85
- end
86
- end
40
+ describe "the SHA1 method" do
41
+ let(:fingerprint_method) { "SHA1" }
87
42
 
88
- describe "fingerprint string with SHA256 alogrithm" do
89
- config <<-CONFIG
90
- filter {
91
- fingerprint {
92
- source => ["clientip"]
93
- key => "longencryptionkey"
94
- method => 'SHA256'
95
- }
96
- }
97
- CONFIG
43
+ it "fingerprints the value" do
44
+ expect(fingerprint).to eq("3a5076c520b4b463f43806896ea0b3978d09dcae")
45
+ end
98
46
 
99
- sample("clientip" => "123.123.123.123") do
100
- insist { subject.get("fingerprint") } == "345bec3eff242d53b568916c2610b3e393d885d6b96d643f38494fd74bf4a9ca"
47
+ context "with HMAC" do
48
+ let(:config) { super().merge("key" => "longencryptionkey") }
49
+
50
+ it "fingerprints the value" do
51
+ expect(fingerprint).to eq("fdc60acc4773dc5ac569ffb78fcb93c9630797f4")
52
+ end
53
+ context "with HMAC and base64 encoding" do
54
+ let(:config) { super().merge("base64encode" => true) }
55
+ it "fingerprints the value" do
56
+ expect(fingerprint).to eq("/cYKzEdz3FrFaf+3j8uTyWMHl/Q=")
57
+ end
58
+ end
59
+ end
60
+ context "and base64 encoding" do
61
+ let(:config) { super().merge("base64encode" => true) }
62
+ it "fingerprints the value" do
63
+ expect(fingerprint).to eq("OlB2xSC0tGP0OAaJbqCzl40J3K4=")
64
+ end
65
+ end
101
66
  end
102
- end
103
67
 
104
- describe "fingerprint string with SHA256 alogrithm and base64 encoding" do
105
- config <<-CONFIG
106
- filter {
107
- fingerprint {
108
- source => ["clientip"]
109
- key => "longencryptionkey"
110
- method => 'SHA256'
111
- base64encode => true
112
- }
113
- }
114
- CONFIG
115
-
116
- sample("clientip" => "123.123.123.123") do
117
- insist { subject.get("fingerprint") } == "NFvsPv8kLVO1aJFsJhCz45PYhda5bWQ/OElP10v0qco="
68
+ context "the SHA256 algorithm" do
69
+ let(:fingerprint_method) { "SHA256" }
70
+ it "fingerprints the value" do
71
+ expect(fingerprint).to eq("4dabcab210766e35f03e77120e6986d6e6d4752b2a9ff22980b9253d026080d8")
72
+ end
73
+ context "with HMAC" do
74
+ let(:config) { super().merge("key" => "longencryptionkey") }
75
+ it "fingerprints the value" do
76
+ expect(fingerprint).to eq("345bec3eff242d53b568916c2610b3e393d885d6b96d643f38494fd74bf4a9ca")
77
+ end
78
+ context "and base64 encoding" do
79
+ let(:config) { super().merge("base64encode" => true) }
80
+ it "fingerprints the value" do
81
+ expect(fingerprint).to eq("NFvsPv8kLVO1aJFsJhCz45PYhda5bWQ/OElP10v0qco=")
82
+ end
83
+ end
84
+ end
118
85
  end
119
- end
120
-
121
- describe "fingerprint string with SHA384 alogrithm" do
122
- config <<-CONFIG
123
- filter {
124
- fingerprint {
125
- source => ["clientip"]
126
- key => "longencryptionkey"
127
- method => 'SHA384'
128
- }
129
- }
130
- CONFIG
131
86
 
132
- sample("clientip" => "123.123.123.123") do
133
- insist { subject.get("fingerprint") } == "22d4c0e8c4fbcdc4887d2038fca7650f0e2e0e2457ff41c06eb2a980dded6749561c814fe182aff93e2538d18593947a"
87
+ context "the SHA384 algorithm" do
88
+ let(:fingerprint_method) { "SHA384" }
89
+ it "fingerprints the value" do
90
+ expect(fingerprint).to eq("fd605b0a3af3e04ce0d7a0b0d9c48d67a12dab811f60072e6eae84e35d567793ffb68a1807536f11c90874065c2a4392")
91
+ end
92
+ context "with HMAC" do
93
+ let(:config) { super().merge("key" => "longencryptionkey") }
94
+ it "fingerprints the value" do
95
+ expect(fingerprint).to eq("22d4c0e8c4fbcdc4887d2038fca7650f0e2e0e2457ff41c06eb2a980dded6749561c814fe182aff93e2538d18593947a")
96
+ end
97
+ context "and base64 encoding" do
98
+ let(:config) { super().merge("base64encode" => true) }
99
+ it "fingerprints the value" do
100
+ expect(fingerprint).to eq("ItTA6MT7zcSIfSA4/KdlDw4uDiRX/0HAbrKpgN3tZ0lWHIFP4YKv+T4lONGFk5R6")
101
+ end
102
+ end
103
+ end
134
104
  end
135
- end
136
-
137
- describe "fingerprint string with SHA384 alogrithm and base64 encoding" do
138
- config <<-CONFIG
139
- filter {
140
- fingerprint {
141
- source => ["clientip"]
142
- key => "longencryptionkey"
143
- method => 'SHA384'
144
- base64encode => true
145
- }
146
- }
147
- CONFIG
148
-
149
- sample("clientip" => "123.123.123.123") do
150
- insist { subject.get("fingerprint") } == "ItTA6MT7zcSIfSA4/KdlDw4uDiRX/0HAbrKpgN3tZ0lWHIFP4YKv+T4lONGFk5R6"
105
+ context "the SHA512 algorithm" do
106
+ let(:fingerprint_method) { "SHA512" }
107
+ it "fingerprints the value" do
108
+ expect(fingerprint).to eq("5468e2dc64ea92b617782aae884b35af60041ac9e168a283615b6a462c54c13d42fa9542cce9b7d76a8124ac6616818905e3e5dd35d6e519f77c3b517558639a")
109
+ end
110
+ context "with HMAC" do
111
+ let(:config) { super().merge("key" => "longencryptionkey") }
112
+ it "fingerprints the value" do
113
+ expect(fingerprint).to eq("11c19b326936c08d6c50a3c847d883e5a1362e6a64dd55201a25f2c1ac1b673f7d8bf15b8f112a4978276d573275e3b14166e17246f670c2a539401c5bfdace8")
114
+ end
115
+ context "and base64 encoding" do
116
+ let(:config) { super().merge("base64encode" => true) }
117
+ it "fingerprints the value" do
118
+ expect(fingerprint).to eq("EcGbMmk2wI1sUKPIR9iD5aE2Lmpk3VUgGiXywawbZz99i/FbjxEqSXgnbVcydeOxQWbhckb2cMKlOUAcW/2s6A==")
119
+ end
120
+ end
121
+ end
151
122
  end
152
- end
153
-
154
- describe "fingerprint string with SHA512 alogrithm" do
155
- config <<-CONFIG
156
- filter {
157
- fingerprint {
158
- source => ["clientip"]
159
- key => "longencryptionkey"
160
- method => 'SHA512'
161
- }
162
- }
163
- CONFIG
164
-
165
- sample("clientip" => "123.123.123.123") do
166
- insist { subject.get("fingerprint") } == "11c19b326936c08d6c50a3c847d883e5a1362e6a64dd55201a25f2c1ac1b673f7d8bf15b8f112a4978276d573275e3b14166e17246f670c2a539401c5bfdace8"
123
+ context "the MD5 algorithm" do
124
+ let(:fingerprint_method) { "MD5" }
125
+ it "fingerprints the value" do
126
+ expect(fingerprint).to eq("ccdd8d3d940a01b2fb3258c059924c0d")
127
+ end
128
+ context "with HMAC" do
129
+ let(:config) { super().merge("key" => "longencryptionkey") }
130
+ it "fingerprints the value" do
131
+ expect(fingerprint).to eq("9336c879e305c9604a3843fc3e75948f")
132
+ end
133
+ context "and base64 encoding" do
134
+ let(:config) { super().merge("base64encode" => true) }
135
+ it "fingerprints the value" do
136
+ expect(fingerprint).to eq("kzbIeeMFyWBKOEP8PnWUjw==")
137
+ end
138
+ end
139
+ end
167
140
  end
168
141
  end
169
142
 
170
- describe "fingerprint string with SHA512 alogrithm and base64 encoding" do
171
- config <<-CONFIG
172
- filter {
173
- fingerprint {
174
- source => ["clientip"]
175
- key => "longencryptionkey"
176
- method => 'SHA512'
177
- base64encode => true
178
- }
179
- }
180
- CONFIG
143
+ context "multiple values in the source field" do
144
+ let(:config) { super().merge("source" => ["clientip" ]) }
145
+ let(:data) { { "clientip" => [ "123.123.123.123", "223.223.223.223" ] } }
181
146
 
182
- sample("clientip" => "123.123.123.123") do
183
- insist { subject.get("fingerprint") } == "EcGbMmk2wI1sUKPIR9iD5aE2Lmpk3VUgGiXywawbZz99i/FbjxEqSXgnbVcydeOxQWbhckb2cMKlOUAcW/2s6A=="
147
+ it "produces a fingerprint array" do
148
+ expect(fingerprint).to eq(["3a5076c520b4b463f43806896ea0b3978d09dcae", "47bbc4e06edebbace047fed35abeceec64968b81"])
184
149
  end
185
150
  end
186
151
 
187
- describe "fingerprint string with MD5 alogrithm" do
188
- config <<-CONFIG
189
- filter {
190
- fingerprint {
191
- source => ["clientip"]
192
- key => "longencryptionkey"
193
- method => 'MD5'
194
- }
195
- }
196
- CONFIG
197
-
198
- sample("clientip" => "123.123.123.123") do
199
- insist { subject.get("fingerprint") } == "9336c879e305c9604a3843fc3e75948f"
152
+ describe "concatenate_all_fields" do
153
+ let(:config) { { "concatenate_all_fields" => true } }
154
+ # The @timestamp field is specified in this sample event as we need the event contents to be constant for the tests
155
+ let(:data) do
156
+ { "@timestamp" => "2017-07-26T14:44:27.064Z", "clientip" => "123.123.123.123", "message" => "This is a test message", "log_level" => "INFO", "offset" => 123456789, "type" => "test" }
200
157
  end
201
- end
202
158
 
203
- describe "fingerprint string with MD5 alogrithm and base64 encoding" do
204
- config <<-CONFIG
205
- filter {
206
- fingerprint {
207
- source => ["clientip"]
208
- key => "longencryptionkey"
209
- method => 'MD5'
210
- base64encode => true
211
- }
212
- }
213
- CONFIG
214
-
215
- sample("clientip" => "123.123.123.123") do
216
- insist { subject.get("fingerprint") } == "kzbIeeMFyWBKOEP8PnWUjw=="
159
+ it "fingerprints the concatenated values" do
160
+ expect(fingerprint).to eq("cbf022518e97860403160ed8a41847c0db104e63")
217
161
  end
218
162
  end
219
163
 
220
- describe "Test field with multiple values" do
221
- config <<-CONFIG
222
- filter {
223
- fingerprint {
224
- source => ["clientip"]
225
- key => "longencryptionkey"
226
- method => 'MD5'
227
- }
228
- }
229
- CONFIG
164
+ context "when multiple fields are used" do
165
+ let(:config) { super().merge("source" => ['field1', 'field2']) }
166
+ let(:data) { { "field1" => "test1", "field2" => "test2" } }
230
167
 
231
- sample("clientip" => [ "123.123.123.123", "223.223.223.223" ]) do
232
- insist { subject.get("fingerprint")} == [ "9336c879e305c9604a3843fc3e75948f", "7a6c66b8d3f42a7d650e3354af508df3" ]
168
+ it "fingerprints the value of the last value" do
169
+ # SHA1 of "test2"
170
+ expect(fingerprint).to eq("109f4b3c50d7b0df729d299bc6f8e9ef9066971f")
233
171
  end
234
- end
235
172
 
236
- describe "Concatenate multiple values into 1" do
237
- config <<-CONFIG
238
- filter {
239
- fingerprint {
240
- source => ['field1', 'field2']
241
- key => "longencryptionkey"
242
- method => 'MD5'
243
- }
244
- }
245
- CONFIG
246
-
247
- sample("field1" => "test1", "field2" => "test2") do
248
- insist { subject.get("fingerprint")} == "872da745e45192c2a1d4bf7c1ff8a370"
173
+ describe "with concatenate_sources" do
174
+ let(:config) { super().merge("concatenate_sources" => true) }
175
+ it "fingerprints the value of concatenated key/pairs" do
176
+ # SHA1 of "|field1|test1|field2|test2|"
177
+ expect(fingerprint).to eq("e3b6b71eedc656f1d29408264e8a75535db985cb")
178
+ end
249
179
  end
250
180
  end
251
181
 
252
182
  describe "PUNCTUATION method" do
253
- config <<-CONFIG
254
- filter {
255
- fingerprint {
256
- source => 'field1'
257
- method => 'PUNCTUATION'
258
- }
259
- }
260
- CONFIG
183
+ let(:fingerprint_method) { 'PUNCTUATION' }
184
+ let(:config) { super().merge("source" => 'field1') }
185
+ let(:data) { { "field1" => "PHP Warning: json_encode() [<a href='function.json-encode'>function.json-encode</a>]: Invalid UTF-8 sequence in argument in /var/www/htdocs/test.php on line 233" } }
261
186
 
262
- sample("field1" => "PHP Warning: json_encode() [<a href='function.json-encode'>function.json-encode</a>]: Invalid UTF-8 sequence in argument in /var/www/htdocs/test.php on line 233") do
263
- insist { subject.get("fingerprint") } == ":_()[<='.-'>.-</>]:-////."
264
- end
265
-
266
- sample("field1" => "Warning: Ruby(ルビ) is an awesome language.") do
267
- insist { subject.get("fingerprint") } == ":()."
187
+ it "extracts punctiation as the fingerprint" do
188
+ expect(fingerprint).to eq(":_()[<='.-'>.-</>]:-////.")
268
189
  end
269
190
  end
270
191
 
271
192
  context 'Timestamps' do
272
193
  epoch_time = Time.at(0).gmtime
194
+ let(:config) { super().merge("source" => ['@timestamp']) }
273
195
 
274
196
  describe 'OpenSSL Fingerprinting' do
275
- config <<-CONFIG
276
- filter {
277
- fingerprint {
278
- source => ['@timestamp']
279
- key => '0123'
280
- method => 'SHA1'
281
- }
282
- }
283
- CONFIG
284
-
285
- sample("@timestamp" => epoch_time) do
286
- insist { subject.get("fingerprint") } == '1d5379ec92d86a67cfc642d55aa050ca312d3b9a'
197
+ let(:config) { super().merge("key" => '0123') }
198
+ let(:fingerprint_method) { "SHA1" }
199
+ let(:data) { { "@timestamp" => epoch_time } }
200
+ it "fingerprints the timestamp correctly" do
201
+ expect(fingerprint).to eq('1d5379ec92d86a67cfc642d55aa050ca312d3b9a')
287
202
  end
288
203
  end
289
204
 
290
205
  describe 'MURMUR3 Fingerprinting' do
291
- config <<-CONFIG
292
- filter {
293
- fingerprint {
294
- source => ['@timestamp']
295
- method => 'MURMUR3'
296
- }
297
- }
298
- CONFIG
299
-
300
- sample("@timestamp" => epoch_time) do
301
- insist { subject.get("fingerprint") } == 743372282
206
+ let(:fingerprint_method) { "MURMUR3" }
207
+ let(:data) { { "@timestamp" => epoch_time } }
208
+ it "fingerprints the timestamp correctly" do
209
+ expect(fingerprint).to eq(743372282)
302
210
  end
303
211
  end
304
212
  end
305
213
 
306
- describe "execution triggers addition of fields and tags" do
307
- config <<-CONFIG
308
- filter {
309
- fingerprint {
310
- source => 'field1'
311
- method => 'PUNCTUATION'
312
- add_field => { 'myfield' => 'myvalue' }
313
- add_tag => ['mytag']
314
- }
214
+ describe "post fingerprint execution triggers" do
215
+ let(:fingerprint_method) { "PUNCTUATION" }
216
+ let(:config) do
217
+ {
218
+ "source" => 'field1',
219
+ "add_field" => { 'myfield' => 'myvalue' },
220
+ "add_tag" => ['mytag']
315
221
  }
316
- CONFIG
222
+ end
223
+ let(:data) { { "field1" => "Hello, World!" } }
317
224
 
318
- sample("field1" => "Hello, World!") do
319
- insist { subject.get("myfield") } == "myvalue"
320
- insist { subject.get("tags") } == ["mytag"]
225
+ it "adds the new field" do
226
+ expect(event.get("myfield")).to eq("myvalue")
227
+ end
228
+ it "adds the new tag" do
229
+ expect(event.get("tags")).to include("mytag")
321
230
  end
322
231
  end
323
232
 
233
+ describe "tolerance to hash order" do
234
+ # insertion order can influence the result of to_hash's keys
235
+ let(:data1) { {
236
+ "a" => {"a0" => 0, "a1" => 1},
237
+ "b" => {"b0" => 0, "b1" => 1},
238
+ } }
239
+ let(:event1) { LogStash::Event.new(data1) }
240
+ let(:data2) { {
241
+ "b" => {"b1" => 1, "b0" => 0},
242
+ "a" => {"a1" => 1, "a0" => 0},
243
+ } }
244
+ let(:event2) { LogStash::Event.new(data2) }
245
+ let(:config) { { "source" => [ "a" ] } }
246
+
247
+ before(:each) do
248
+ # for testing purposes we want to ensure the hash order is different.
249
+ # since we can't easily control the order on the underlying Map,
250
+ # we're mocking the order here:
251
+ allow(event1).to receive(:to_hash).and_return(data1)
252
+ allow(event2).to receive(:to_hash).and_return(data2)
253
+ # by default event.get(key) fetches data from the event.
254
+ # mocking the default value has to be done first, and only
255
+ # then we can mock the getting "a" and "b"
256
+ allow(event1).to receive(:get).and_call_original
257
+ allow(event2).to receive(:get).and_call_original
258
+ # mock event.get("a") and event.get("b") for both events
259
+ # so we can inject an inconsistent order for the tests
260
+ allow(event1).to receive(:get).with("a") {|arg| data1["a"] }
261
+ allow(event1).to receive(:get).with("b") {|arg| data1["b"] }
262
+ allow(event2).to receive(:get).with("a") {|arg| data2["a"] }
263
+ allow(event2).to receive(:get).with("b") {|arg| data2["b"] }
264
+ plugin.filter(event1)
265
+ plugin.filter(event2)
266
+ end
267
+ it "computes the same hash" do
268
+ # confirm the order of the keys in the nested hash is different
269
+ # (of course it is, we just mocked the to_hash return)
270
+ expect(event1.to_hash["a"].keys).to_not eq(event2.to_hash["a"].keys)
271
+ # let's check that the fingerprint doesn't care about the insertion order
272
+ expect(event1.get("fingerprint")).to eq(event2.get("fingerprint"))
273
+ end
274
+ context "concatenate_sources" do
275
+ let("config") { { "source" => [ "a", "b"], "concatenate_sources" => true } }
276
+ it "computes the same hash" do
277
+ expect(event1.get("fingerprint")).to eq(event2.get("fingerprint"))
278
+ end
279
+ end
280
+ context "concatenate_all_fields => true" do
281
+ let(:config) { { "concatenate_all_fields" => true } }
282
+ it "computes the same hash" do
283
+ expect(event1.get("fingerprint")).to eq(event2.get("fingerprint"))
284
+ end
285
+ end
286
+ end
324
287
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-filter-fingerprint
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.1.1
4
+ version: 3.2.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2017-08-15 00:00:00.000000000 Z
11
+ date: 2021-03-11 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -58,7 +58,9 @@ dependencies:
58
58
  - - ">="
59
59
  - !ruby/object:Gem::Version
60
60
  version: '0'
61
- description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
61
+ description: This gem is a Logstash plugin required to be installed on top of the
62
+ Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
63
+ gem is not a stand-alone program
62
64
  email: info@elastic.co
63
65
  executables: []
64
66
  extensions: []
@@ -76,7 +78,7 @@ files:
76
78
  - spec/filters/fingerprint_spec.rb
77
79
  homepage: http://www.elastic.co/guide/en/logstash/current/index.html
78
80
  licenses:
79
- - Apache License (2.0)
81
+ - Apache-2.0
80
82
  metadata:
81
83
  logstash_plugin: 'true'
82
84
  logstash_group: filter
@@ -96,9 +98,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
96
98
  version: '0'
97
99
  requirements: []
98
100
  rubyforge_project:
99
- rubygems_version: 2.4.8
101
+ rubygems_version: 2.6.13
100
102
  signing_key:
101
103
  specification_version: 4
102
- summary: Fingerprint fields using by replacing values with a consistent hash.
104
+ summary: Fingerprints fields by replacing values with a consistent hash
103
105
  test_files:
104
106
  - spec/filters/fingerprint_spec.rb