logstash-filter-railsroutes 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ab10411e82a44e2f30ca10dd5f01ec35443205ea
+  data.tar.gz: 1bf469c3df7a3e4e0e8bbd2e5964c247eeea148e
+SHA512:
+  metadata.gz: 25519f42cecfe2ada79ebc88858cde419e4353f1e6a4bf1a9493164a4dee1b90cdaf06b46744e9fa13b562c708efdee8405e845fdc06597a238535ead62183f0
+  data.tar.gz: a8264100463c758a7aa09b2f3bea3f99a3ec84231602ef6efb189cb09b6903bbca9bf87a7c61e6a60479c144ca14ba917ddacd16af24ba5520c1026c20868b34

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+## 0.1.0 (2015/11/27)
+=====================
+- Make use of rails routes to recognize controller/action information inside URI.

data/CONTRIBUTORS

@@ -0,0 +1,11 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Pier-Hugues Pellerin (ph)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-filter-example
+Example filter plugin. This should help bootstrap your effort to write your own filter plugin!

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+gem 'journey', '1.0.4'

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,68 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+The aim of this filter is to recognize a URI in rails `controller#action` form.
+
+### Prerequisite
+- rails routes table.
+Make use of rails rake `routes`, we can gather all path patterns in a rails application.
+`bundle exec rake routes | tail -n +2 > routes_spec`
+
+The result may look like
+```
+users POST /users(.:format)     users#create
+      GET  /users(.:format)     users#index
+      GET  /users/:id(.:format) users#show
+      ...
+```
+
+### Example #1
+- with these given logs:
+```
+2015-11-18T09:45:58.797031Z "GET https://some.domain.com/api/users/1"
+```
+
+- you can use `grok` to recognize http verb and uri separately and use `railsroutes` to figure out the rails controller and action, even the parameters inside the uri can be extracted as well.
+```
+filter {
+  grok {
+    match => ['message', '%{TIMESTAMP_ISO8601:timestamp} "%{WORD:http_verb} %{URI:url}"']
+  }
+  railsroutes {
+    verb_source => 'http_verb'
+    uri_source => 'url'
+    routes_spec => '/somewhere/to/your/rails/routes/table'
+    api_prefix => 'https://some.domain.com/api'
+    target => 'rails'
+  }
+}
+```
+
+- the final event then looks like:
+```json
+{
+    "message": "2015-11-18T09:45:58.797031Z \"GET https://some.domain.com/api/users/1\"",
+    "http_verb": "GET",
+    "url": "https://some.domain.com/api/users/1",
+    "rails": {
+        "controller#action": "users#show",
+        "id": "1",
+        "format": null
+    }
+}
+```
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/filters/railsroutes.rb

@@ -0,0 +1,117 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require 'journey'
+
+class LogStash::Filters::RailsRoutes < LogStash::Filters::Base
+  config_name "railsroutes"
+
+  config :routes_spec, :validate => :string, :required => true
+  config :verb_source, :validate => :string, :required => true
+  config :uri_source, :validate => :string, :required => true
+  config :api_prefix, :validate => :string, :default => ''
+  config :target, :validate => :string
+
+  class RoutePattern
+    ParseError = Class.new StandardError
+
+    attr_reader :controller_action
+
+    def self.parse_spec_line(line)
+      sub_strings = line.split
+
+      case sub_strings.size
+      when 3
+        verb, pattern, controller_action = sub_strings
+        self.new(verb, pattern, controller_action)
+      when 4
+        _, verb, pattern, controller_action = sub_strings
+        self.new(verb, pattern, controller_action)
+      else
+        fail ParseError, "Cannot parse route spec: #{line}"
+      end
+    end
+
+    def initialize(verb, path, controller_action)
+      if verb.include? '|'
+        @verbs = verb.split('|').map(&:upcase)
+      else
+        @verbs = [verb.upcase]
+      end
+
+      @path_pattern = Journey::Path::Pattern.new(path)
+      @controller_action = controller_action
+    end
+
+    def match(verb, uri)
+      return nil unless @verbs.include?(verb)
+      @path_pattern =~ uri
+    end
+  end
+
+  public
+  def register
+    File.open(@routes_spec) do |f|
+      @patterns = f.each_line.map do |line|
+        begin
+          RoutePattern.parse_spec_line(line)
+        rescue RoutePattern::ParseError => err
+          @logger.error(err.message)
+          nil
+        end
+      end
+
+      @patterns.compact!
+    end
+  end # def register
+
+  public
+  def filter(event)
+    verb = event[@verb_source]
+    uri = event[@uri_source]
+    target = @target ? (event[@target] ||= {}) : event
+    target['controller#action'] = nil
+
+    if verb.nil? || uri.nil?
+      @logger.error("Incomplete source: verb = '#{verb}', uri = '#{uri}'")
+      return
+    end
+
+    verb = verb.upcase
+    uri = normalize_uri(uri)
+    match(verb, uri, target)
+
+    unless target['controller#action']
+      @logger.warn("Unrecognizable: #{verb} #{uri}")
+    end
+
+    # filter_matched should go in the last line of our successful code
+    filter_matched(event)
+  end # def filter
+
+  private
+  def normalize_uri(uri)
+    if @api_prefix != ''
+      if uri.start_with? @api_prefix
+        uri = uri[@api_prefix.size..-1]
+        uri = '/' + uri unless uri.start_with? '/'
+      else
+        @logger.warn("'#{uri}' does not start with '#{@api_prefix}'")
+        return uri
+      end
+    end
+
+    uri = uri[0...-1] if uri.end_with? '/'
+    uri.gsub(/\/\//, '/')
+  end
+
+  private
+  def match(verb, uri, target)
+    @patterns.each do |pattern|
+      next unless result = pattern.match(verb, uri)
+      result.names.each_with_index { |name, ix| target[name] = result[ix + 1] }
+      target['controller#action'] = pattern.controller_action
+      break
+    end
+  end
+end # class LogStash::Filters::RailsRoutes

data/logstash-filter-railsroutes.gemspec

@@ -0,0 +1,36 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-filter-railsroutes'
+  s.version = '0.1.0'
+  s.licenses = ['Apache License (2.0)']
+  s.summary = 'This example uses rails route to match URI in logstash'
+  s.description = 'This gem is a logstash plugin required to be installed on \
+top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. \
+This gem is not a stand-alone program.'
+  s.authors = ['Yu Liang']
+  s.email = 'yu.liang@thekono.com'
+  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir[
+    'lib/**/*',
+    'spec/**/*',
+    'vendor/**/*',
+    '*.gemspec',
+    '*.md',
+    'CONTRIBUTORS',
+    'Gemfile',
+    'LICENSE',
+    'NOTICE.TXT'
+  ]
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
+  s.add_runtime_dependency 'journey', '1.0.4'
+  s.add_development_dependency 'logstash-devutils'
+end

data/spec/filters/railsroutes_spec.rb

@@ -0,0 +1,147 @@
+# encoding: utf-8
+require 'spec_helper'
+require "logstash/filters/railsroutes"
+
+describe LogStash::Filters::RailsRoutes do
+  subject { LogStash::Filters::RailsRoutes.new(config) }
+
+  let(:attrs) { Hash[] }
+  let(:event) { LogStash::Event.new(attrs) }
+  let(:config) do
+    {
+      'verb_source' => 'verb',
+      'uri_source' => 'uri',
+      'routes_spec' => @route_spec_filename
+    }
+  end
+
+  before(:each) do
+    file = Tempfile.new('railsroutes')
+    file.write(routes_spec_content)
+    file.close
+    @route_spec_filename = file.path
+  end
+
+  after(:each) do
+    File.exists?(@route_spec_filename) && File.unlink(@route_spec_filename)
+  end
+
+  describe 'routes spec content' do
+    before(:each) do
+      subject.register()
+    end
+
+    context 'when prefix exists in routes spec' do
+      let(:attrs) { Hash['verb', 'GET', 'uri', '/resources/1'] }
+      let(:routes_spec_content) do <<-SPEC
+        resources GET /resources/:id(.:format) resources#show
+      SPEC
+      end
+
+      it 'can match' do
+        subject.filter(event)
+        expect(event['controller#action']).to eq 'resources#show'
+        expect(event['id']).to eq '1'
+        expect(event['format']).to be_nil
+      end
+    end
+
+    context 'when prefix does not exist in routes spec' do
+      let(:attrs) { Hash['verb', 'GET', 'uri', '/resources/1'] }
+      let(:routes_spec_content) do <<-SPEC
+        GET /resources/:id(.:format) resources#show
+      SPEC
+      end
+
+      it 'can match' do
+        subject.filter(event)
+        expect(event['controller#action']).to eq 'resources#show'
+        expect(event['id']).to eq '1'
+        expect(event['format']).to be_nil
+      end
+    end
+  end
+
+  describe 'api_prefix' do
+    let(:api_prefix) { 'https://api.domain.com/' }
+    let(:attrs) do
+      {
+        'verb' => 'GET',
+        'uri' => "#{api_prefix}resources/1"
+      }
+    end
+    let(:routes_spec_content) do <<-SPEC
+      resources GET /resources/:id(.:format) resources#show
+    SPEC
+    end
+
+    before(:each) do
+      config['api_prefix'] = api_prefix
+      subject.register()
+    end
+
+    it 'can match the part after api prefix' do
+      subject.filter(event)
+      expect(event['controller#action']).to eq 'resources#show'
+      expect(event['id']).to eq '1'
+      expect(event['format']).to be_nil
+    end
+  end
+
+  describe 'target' do
+    let(:attrs) do
+      {'verb' => 'GET', 'uri' => "/resources/1"}
+    end
+    let(:routes_spec_content) do <<-SPEC
+      resources GET /resources/:id(.:format) resources#show
+    SPEC
+    end
+
+    before(:each) do
+      config['target'] = 'target'
+      subject.register()
+    end
+
+    it 'can match the part after api prefix' do
+      subject.filter(event)
+      obj = event['target']
+      expect(obj['controller#action']).to eq 'resources#show'
+      expect(obj['id']).to eq '1'
+      expect(obj['format']).to be_nil
+    end
+  end
+
+  describe 'normalize uri' do
+    let(:routes_spec_content) do <<-SPEC
+      resources POST /resources(.:format)     resources#create
+                GET  /resources/:id(.:format) resources#show
+    SPEC
+    end
+
+    before(:each) do
+      subject.register()
+    end
+
+    context 'when uri ends with a slash' do
+      let(:attrs) do
+        {'verb' => 'POST', 'uri' => '/resources/'}
+      end
+
+      it 'can match' do
+        subject.filter(event)
+        expect(event['controller#action']).to eq 'resources#create'
+      end
+    end
+
+    context 'when uri has double slash' do
+      let(:attrs) do
+        {'verb' => 'GET', 'uri' => '//resources//1'}
+      end
+
+      it 'can match' do
+        subject.filter(event)
+        expect(event['controller#action']).to eq 'resources#show'
+      end
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,2 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,109 @@
+--- !ruby/object:Gem::Specification
+name: logstash-filter-railsroutes
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Yu Liang
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2015-11-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  prerelease: false
+  type: :runtime
+- !ruby/object:Gem::Dependency
+  name: journey
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.0.4
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.0.4
+  prerelease: false
+  type: :runtime
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :development
+description: |-
+  This gem is a logstash plugin required to be installed on \
+  top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. \
+  This gem is not a stand-alone program.
+email: yu.liang@thekono.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/logstash/filters/railsroutes.rb
+- spec/spec_helper.rb
+- spec/filters/railsroutes_spec.rb
+- logstash-filter-railsroutes.gemspec
+- CHANGELOG.md
+- DEVELOPER.md
+- README.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: filter
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.1.9
+signing_key:
+specification_version: 4
+summary: This example uses rails route to match URI in logstash
+test_files:
+- spec/spec_helper.rb
+- spec/filters/railsroutes_spec.rb