logstash-filter-railsroutes 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: ab10411e82a44e2f30ca10dd5f01ec35443205ea
4
+ data.tar.gz: 1bf469c3df7a3e4e0e8bbd2e5964c247eeea148e
5
+ SHA512:
6
+ metadata.gz: 25519f42cecfe2ada79ebc88858cde419e4353f1e6a4bf1a9493164a4dee1b90cdaf06b46744e9fa13b562c708efdee8405e845fdc06597a238535ead62183f0
7
+ data.tar.gz: a8264100463c758a7aa09b2f3bea3f99a3ec84231602ef6efb189cb09b6903bbca9bf87a7c61e6a60479c144ca14ba917ddacd16af24ba5520c1026c20868b34
@@ -0,0 +1,3 @@
1
+ ## 0.1.0 (2015/11/27)
2
+ =====================
3
+ - Make use of rails routes to recognize controller/action information inside URI.
@@ -0,0 +1,11 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * Aaron Mildenstein (untergeek)
6
+ * Pier-Hugues Pellerin (ph)
7
+
8
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
9
+ Logstash, and you aren't on the list above and want to be, please let us know
10
+ and we'll make sure you're here. Contributions from folks like you are what make
11
+ open source awesome.
@@ -0,0 +1,2 @@
1
+ # logstash-filter-example
2
+ Example filter plugin. This should help bootstrap your effort to write your own filter plugin!
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
3
+ gem 'journey', '1.0.4'
data/LICENSE ADDED
@@ -0,0 +1,13 @@
1
+ Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
+ limitations under the License.
@@ -0,0 +1,5 @@
1
+ Elasticsearch
2
+ Copyright 2012-2015 Elasticsearch
3
+
4
+ This product includes software developed by The Apache Software
5
+ Foundation (http://www.apache.org/).
@@ -0,0 +1,68 @@
1
+ # Logstash Plugin
2
+
3
+ This is a plugin for [Logstash](https://github.com/elastic/logstash).
4
+
5
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
6
+
7
+ ## Documentation
8
+
9
+ The aim of this filter is to recognize a URI in rails `controller#action` form.
10
+
11
+ ### Prerequisite
12
+ - rails routes table.
13
+ Make use of rails rake `routes`, we can gather all path patterns in a rails application.
14
+ `bundle exec rake routes | tail -n +2 > routes_spec`
15
+
16
+ The result may look like
17
+ ```
18
+ users POST /users(.:format) users#create
19
+ GET /users(.:format) users#index
20
+ GET /users/:id(.:format) users#show
21
+ ...
22
+ ```
23
+
24
+ ### Example #1
25
+ - with these given logs:
26
+ ```
27
+ 2015-11-18T09:45:58.797031Z "GET https://some.domain.com/api/users/1"
28
+ ```
29
+
30
+ - you can use `grok` to recognize http verb and uri separately and use `railsroutes` to figure out the rails controller and action, even the parameters inside the uri can be extracted as well.
31
+ ```
32
+ filter {
33
+ grok {
34
+ match => ['message', '%{TIMESTAMP_ISO8601:timestamp} "%{WORD:http_verb} %{URI:url}"']
35
+ }
36
+ railsroutes {
37
+ verb_source => 'http_verb'
38
+ uri_source => 'url'
39
+ routes_spec => '/somewhere/to/your/rails/routes/table'
40
+ api_prefix => 'https://some.domain.com/api'
41
+ target => 'rails'
42
+ }
43
+ }
44
+ ```
45
+
46
+ - the final event then looks like:
47
+ ```json
48
+ {
49
+ "message": "2015-11-18T09:45:58.797031Z \"GET https://some.domain.com/api/users/1\"",
50
+ "http_verb": "GET",
51
+ "url": "https://some.domain.com/api/users/1",
52
+ "rails": {
53
+ "controller#action": "users#show",
54
+ "id": "1",
55
+ "format": null
56
+ }
57
+ }
58
+ ```
59
+
60
+ ## Contributing
61
+
62
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
63
+
64
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
65
+
66
+ It is more important to the community that you are able to contribute.
67
+
68
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
@@ -0,0 +1,117 @@
1
+ # encoding: utf-8
2
+ require "logstash/filters/base"
3
+ require "logstash/namespace"
4
+ require 'journey'
5
+
6
+ class LogStash::Filters::RailsRoutes < LogStash::Filters::Base
7
+ config_name "railsroutes"
8
+
9
+ config :routes_spec, :validate => :string, :required => true
10
+ config :verb_source, :validate => :string, :required => true
11
+ config :uri_source, :validate => :string, :required => true
12
+ config :api_prefix, :validate => :string, :default => ''
13
+ config :target, :validate => :string
14
+
15
+ class RoutePattern
16
+ ParseError = Class.new StandardError
17
+
18
+ attr_reader :controller_action
19
+
20
+ def self.parse_spec_line(line)
21
+ sub_strings = line.split
22
+
23
+ case sub_strings.size
24
+ when 3
25
+ verb, pattern, controller_action = sub_strings
26
+ self.new(verb, pattern, controller_action)
27
+ when 4
28
+ _, verb, pattern, controller_action = sub_strings
29
+ self.new(verb, pattern, controller_action)
30
+ else
31
+ fail ParseError, "Cannot parse route spec: #{line}"
32
+ end
33
+ end
34
+
35
+ def initialize(verb, path, controller_action)
36
+ if verb.include? '|'
37
+ @verbs = verb.split('|').map(&:upcase)
38
+ else
39
+ @verbs = [verb.upcase]
40
+ end
41
+
42
+ @path_pattern = Journey::Path::Pattern.new(path)
43
+ @controller_action = controller_action
44
+ end
45
+
46
+ def match(verb, uri)
47
+ return nil unless @verbs.include?(verb)
48
+ @path_pattern =~ uri
49
+ end
50
+ end
51
+
52
+ public
53
+ def register
54
+ File.open(@routes_spec) do |f|
55
+ @patterns = f.each_line.map do |line|
56
+ begin
57
+ RoutePattern.parse_spec_line(line)
58
+ rescue RoutePattern::ParseError => err
59
+ @logger.error(err.message)
60
+ nil
61
+ end
62
+ end
63
+
64
+ @patterns.compact!
65
+ end
66
+ end # def register
67
+
68
+ public
69
+ def filter(event)
70
+ verb = event[@verb_source]
71
+ uri = event[@uri_source]
72
+ target = @target ? (event[@target] ||= {}) : event
73
+ target['controller#action'] = nil
74
+
75
+ if verb.nil? || uri.nil?
76
+ @logger.error("Incomplete source: verb = '#{verb}', uri = '#{uri}'")
77
+ return
78
+ end
79
+
80
+ verb = verb.upcase
81
+ uri = normalize_uri(uri)
82
+ match(verb, uri, target)
83
+
84
+ unless target['controller#action']
85
+ @logger.warn("Unrecognizable: #{verb} #{uri}")
86
+ end
87
+
88
+ # filter_matched should go in the last line of our successful code
89
+ filter_matched(event)
90
+ end # def filter
91
+
92
+ private
93
+ def normalize_uri(uri)
94
+ if @api_prefix != ''
95
+ if uri.start_with? @api_prefix
96
+ uri = uri[@api_prefix.size..-1]
97
+ uri = '/' + uri unless uri.start_with? '/'
98
+ else
99
+ @logger.warn("'#{uri}' does not start with '#{@api_prefix}'")
100
+ return uri
101
+ end
102
+ end
103
+
104
+ uri = uri[0...-1] if uri.end_with? '/'
105
+ uri.gsub(/\/\//, '/')
106
+ end
107
+
108
+ private
109
+ def match(verb, uri, target)
110
+ @patterns.each do |pattern|
111
+ next unless result = pattern.match(verb, uri)
112
+ result.names.each_with_index { |name, ix| target[name] = result[ix + 1] }
113
+ target['controller#action'] = pattern.controller_action
114
+ break
115
+ end
116
+ end
117
+ end # class LogStash::Filters::RailsRoutes
@@ -0,0 +1,36 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-filter-railsroutes'
3
+ s.version = '0.1.0'
4
+ s.licenses = ['Apache License (2.0)']
5
+ s.summary = 'This example uses rails route to match URI in logstash'
6
+ s.description = 'This gem is a logstash plugin required to be installed on \
7
+ top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. \
8
+ This gem is not a stand-alone program.'
9
+ s.authors = ['Yu Liang']
10
+ s.email = 'yu.liang@thekono.com'
11
+ s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
12
+ s.require_paths = ["lib"]
13
+
14
+ # Files
15
+ s.files = Dir[
16
+ 'lib/**/*',
17
+ 'spec/**/*',
18
+ 'vendor/**/*',
19
+ '*.gemspec',
20
+ '*.md',
21
+ 'CONTRIBUTORS',
22
+ 'Gemfile',
23
+ 'LICENSE',
24
+ 'NOTICE.TXT'
25
+ ]
26
+ # Tests
27
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
28
+
29
+ # Special flag to let us know this is actually a logstash plugin
30
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
31
+
32
+ # Gem dependencies
33
+ s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
34
+ s.add_runtime_dependency 'journey', '1.0.4'
35
+ s.add_development_dependency 'logstash-devutils'
36
+ end
@@ -0,0 +1,147 @@
1
+ # encoding: utf-8
2
+ require 'spec_helper'
3
+ require "logstash/filters/railsroutes"
4
+
5
+ describe LogStash::Filters::RailsRoutes do
6
+ subject { LogStash::Filters::RailsRoutes.new(config) }
7
+
8
+ let(:attrs) { Hash[] }
9
+ let(:event) { LogStash::Event.new(attrs) }
10
+ let(:config) do
11
+ {
12
+ 'verb_source' => 'verb',
13
+ 'uri_source' => 'uri',
14
+ 'routes_spec' => @route_spec_filename
15
+ }
16
+ end
17
+
18
+ before(:each) do
19
+ file = Tempfile.new('railsroutes')
20
+ file.write(routes_spec_content)
21
+ file.close
22
+ @route_spec_filename = file.path
23
+ end
24
+
25
+ after(:each) do
26
+ File.exists?(@route_spec_filename) && File.unlink(@route_spec_filename)
27
+ end
28
+
29
+ describe 'routes spec content' do
30
+ before(:each) do
31
+ subject.register()
32
+ end
33
+
34
+ context 'when prefix exists in routes spec' do
35
+ let(:attrs) { Hash['verb', 'GET', 'uri', '/resources/1'] }
36
+ let(:routes_spec_content) do <<-SPEC
37
+ resources GET /resources/:id(.:format) resources#show
38
+ SPEC
39
+ end
40
+
41
+ it 'can match' do
42
+ subject.filter(event)
43
+ expect(event['controller#action']).to eq 'resources#show'
44
+ expect(event['id']).to eq '1'
45
+ expect(event['format']).to be_nil
46
+ end
47
+ end
48
+
49
+ context 'when prefix does not exist in routes spec' do
50
+ let(:attrs) { Hash['verb', 'GET', 'uri', '/resources/1'] }
51
+ let(:routes_spec_content) do <<-SPEC
52
+ GET /resources/:id(.:format) resources#show
53
+ SPEC
54
+ end
55
+
56
+ it 'can match' do
57
+ subject.filter(event)
58
+ expect(event['controller#action']).to eq 'resources#show'
59
+ expect(event['id']).to eq '1'
60
+ expect(event['format']).to be_nil
61
+ end
62
+ end
63
+ end
64
+
65
+ describe 'api_prefix' do
66
+ let(:api_prefix) { 'https://api.domain.com/' }
67
+ let(:attrs) do
68
+ {
69
+ 'verb' => 'GET',
70
+ 'uri' => "#{api_prefix}resources/1"
71
+ }
72
+ end
73
+ let(:routes_spec_content) do <<-SPEC
74
+ resources GET /resources/:id(.:format) resources#show
75
+ SPEC
76
+ end
77
+
78
+ before(:each) do
79
+ config['api_prefix'] = api_prefix
80
+ subject.register()
81
+ end
82
+
83
+ it 'can match the part after api prefix' do
84
+ subject.filter(event)
85
+ expect(event['controller#action']).to eq 'resources#show'
86
+ expect(event['id']).to eq '1'
87
+ expect(event['format']).to be_nil
88
+ end
89
+ end
90
+
91
+ describe 'target' do
92
+ let(:attrs) do
93
+ {'verb' => 'GET', 'uri' => "/resources/1"}
94
+ end
95
+ let(:routes_spec_content) do <<-SPEC
96
+ resources GET /resources/:id(.:format) resources#show
97
+ SPEC
98
+ end
99
+
100
+ before(:each) do
101
+ config['target'] = 'target'
102
+ subject.register()
103
+ end
104
+
105
+ it 'can match the part after api prefix' do
106
+ subject.filter(event)
107
+ obj = event['target']
108
+ expect(obj['controller#action']).to eq 'resources#show'
109
+ expect(obj['id']).to eq '1'
110
+ expect(obj['format']).to be_nil
111
+ end
112
+ end
113
+
114
+ describe 'normalize uri' do
115
+ let(:routes_spec_content) do <<-SPEC
116
+ resources POST /resources(.:format) resources#create
117
+ GET /resources/:id(.:format) resources#show
118
+ SPEC
119
+ end
120
+
121
+ before(:each) do
122
+ subject.register()
123
+ end
124
+
125
+ context 'when uri ends with a slash' do
126
+ let(:attrs) do
127
+ {'verb' => 'POST', 'uri' => '/resources/'}
128
+ end
129
+
130
+ it 'can match' do
131
+ subject.filter(event)
132
+ expect(event['controller#action']).to eq 'resources#create'
133
+ end
134
+ end
135
+
136
+ context 'when uri has double slash' do
137
+ let(:attrs) do
138
+ {'verb' => 'GET', 'uri' => '//resources//1'}
139
+ end
140
+
141
+ it 'can match' do
142
+ subject.filter(event)
143
+ expect(event['controller#action']).to eq 'resources#show'
144
+ end
145
+ end
146
+ end
147
+ end
@@ -0,0 +1,2 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
metadata ADDED
@@ -0,0 +1,109 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-filter-railsroutes
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Yu Liang
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-11-27 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: logstash-core
15
+ version_requirements: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - '>='
18
+ - !ruby/object:Gem::Version
19
+ version: 2.0.0
20
+ - - <
21
+ - !ruby/object:Gem::Version
22
+ version: 3.0.0
23
+ requirement: !ruby/object:Gem::Requirement
24
+ requirements:
25
+ - - '>='
26
+ - !ruby/object:Gem::Version
27
+ version: 2.0.0
28
+ - - <
29
+ - !ruby/object:Gem::Version
30
+ version: 3.0.0
31
+ prerelease: false
32
+ type: :runtime
33
+ - !ruby/object:Gem::Dependency
34
+ name: journey
35
+ version_requirements: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - '='
38
+ - !ruby/object:Gem::Version
39
+ version: 1.0.4
40
+ requirement: !ruby/object:Gem::Requirement
41
+ requirements:
42
+ - - '='
43
+ - !ruby/object:Gem::Version
44
+ version: 1.0.4
45
+ prerelease: false
46
+ type: :runtime
47
+ - !ruby/object:Gem::Dependency
48
+ name: logstash-devutils
49
+ version_requirements: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - '>='
52
+ - !ruby/object:Gem::Version
53
+ version: '0'
54
+ requirement: !ruby/object:Gem::Requirement
55
+ requirements:
56
+ - - '>='
57
+ - !ruby/object:Gem::Version
58
+ version: '0'
59
+ prerelease: false
60
+ type: :development
61
+ description: |-
62
+ This gem is a logstash plugin required to be installed on \
63
+ top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. \
64
+ This gem is not a stand-alone program.
65
+ email: yu.liang@thekono.com
66
+ executables: []
67
+ extensions: []
68
+ extra_rdoc_files: []
69
+ files:
70
+ - lib/logstash/filters/railsroutes.rb
71
+ - spec/spec_helper.rb
72
+ - spec/filters/railsroutes_spec.rb
73
+ - logstash-filter-railsroutes.gemspec
74
+ - CHANGELOG.md
75
+ - DEVELOPER.md
76
+ - README.md
77
+ - CONTRIBUTORS
78
+ - Gemfile
79
+ - LICENSE
80
+ - NOTICE.TXT
81
+ homepage: http://www.elastic.co/guide/en/logstash/current/index.html
82
+ licenses:
83
+ - Apache License (2.0)
84
+ metadata:
85
+ logstash_plugin: 'true'
86
+ logstash_group: filter
87
+ post_install_message:
88
+ rdoc_options: []
89
+ require_paths:
90
+ - lib
91
+ required_ruby_version: !ruby/object:Gem::Requirement
92
+ requirements:
93
+ - - '>='
94
+ - !ruby/object:Gem::Version
95
+ version: '0'
96
+ required_rubygems_version: !ruby/object:Gem::Requirement
97
+ requirements:
98
+ - - '>='
99
+ - !ruby/object:Gem::Version
100
+ version: '0'
101
+ requirements: []
102
+ rubyforge_project:
103
+ rubygems_version: 2.1.9
104
+ signing_key:
105
+ specification_version: 4
106
+ summary: This example uses rails route to match URI in logstash
107
+ test_files:
108
+ - spec/spec_helper.rb
109
+ - spec/filters/railsroutes_spec.rb