logstash-filter-fix_protocol 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 01e4c787451d2df8a4d094719f6773f41dc32b48
+  data.tar.gz: 59d92aec02f3566804710507b42e046814817100
+SHA512:
+  metadata.gz: 1c20c43d362899f2423da2799621d48c5af3f9c2775313a33ddac837ca68d93098b34ed098aa2a214235f0d931a0aa615699a5c22651fd455ff2ef16cb0681fb
+  data.tar.gz: 27940ad87626f23d13428a78fc243ede4f453551d35404f5af3592a8ec75b508aaaf494769b0a108424864ce0c1db1d7bbf8be701661e60654ff287c39d98c09

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,13 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, age, or religion.
+
+Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.
+
+This Code of Conduct is adapted from the [Contributor Covenant](http:contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)

data/CONTRIBUTING.md

@@ -0,0 +1,34 @@
+# Contributing to FIX Message Logstash Filter
+
+Thanks for taking the time to contribute to FIX Message Logstash Filter!
+
+If you come across any issues, please [tell us](https://github.com/connamara/logstash-filter-fix_protocol/issues).
+Pull requests (with tests) are appreciated. No pull request is too small. Please help with:
+
+* Reporting bugs
+* Suggesting features
+* Writing or improving documentation
+* Fixing typos
+* Cleaning whitespace
+* Refactoring code
+* Adding tests
+* Closing [issues](https://github.com/connamara/logstash-filter-fix_protocol/issues)
+
+
+## Before you contribute
+
+* Did you find a bug?  **Ensure the bug was not already reported** by searching on Github under [Issues](https://github.com/connamara/logstash-filter-fix_protocol/issues).
+
+## Steps to Contribute:
+
+1. Fork the [official repository](https://github.com/connamara/logstash-filter-fix_protocol/).
+2. Make your changes in a topic branch.
+3. Send a pull request.
+4. If you are new contributor to this project, sign the CLA
+
+### Notes:
+
+* If you report a bug and don't include a fix, please include a failing test.
+* Contributions without a signed CLA won't be accepted.
+* Generally, contributions without tests won't be accepted.
+* Contributions that fail the automated build won't be accepted.

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fix_logstash_plugin.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,13 @@
+Copyright 2016 Connamara Systems, llc
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/QUICKFIX_LICENSE.txt

@@ -0,0 +1,46 @@
+The QuickFIX Software License, Version 1.0
+ 
+Copyright (c) 2001-2010 quickfixengine.org  All rights
+reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+ 
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in
+   the documentation and/or other materials provided with the
+   distribution.
+
+3. The end-user documentation included with the redistribution,
+   if any, must include the following acknowledgment:
+      "This product includes software developed by
+       quickfixengine.org (http://www.quickfixengine.org/)."
+   Alternately, this acknowledgment may appear in the software itself,
+   if and wherever such third-party acknowledgments normally appear.
+ 
+4. The names "QuickFIX" and "quickfixengine.org" must
+   not be used to endorse or promote products derived from this
+   software without prior written permission. For written
+   permission, please contact ask@quickfixengine.org
+ 
+5. Products derived from this software may not be called "QuickFIX",
+   nor may "QuickFIX" appear in their name, without prior written
+   permission of quickfixengine.org
+ 
+THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
+WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED.  IN NO EVENT SHALL QUICKFIXENGINE.ORG OR
+ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.
+

data/README.md

@@ -0,0 +1,151 @@
+# FIX Protocol Logstash Filter [![Build Status](https://travis-ci.org/connamara/logstash-filter-fix_protocol.svg?branch=master)](https://travis-ci.org/connamara/logstash-filter-fix_protocol)
+
+A LogStash filter plugin for FIX Message parsing
+
+Given a FIX log file that looks like this:
+
+```
+2015-08-26 23:08:38,096 FIX.4.2:DUMMY_INC->ANOTHER_INC: 8=FIX.4.2
9=184
35=F
34=2
49=ANOTHER_INC
50=DefaultSenderSubID
52=20150826-23:08:38.094
56=DUMMY_INC
1=DefaultAccount
11=clordid_of_cancel
41=151012569
54=1
55=ITER
60=20250407-13:14:15
167=FUT
200=201512
10=147

+2015-08-31 17:48:20,890 FIXT.1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT.1.1
9=140
35=W
34=2
49=DUMMY_INC
52=20150831-17:48:20.890
56=ANOTHER_INC
22=99
48=.AQUA-W
262=golden_path_test
268=1
269=3
270=640754
272=20150831
273=17:48:20.882
10=070

+2015-08-31 20:48:26,536 FIXT.1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT.1.1
9=189
35=W
34=5
49=DUMMY_INC
52=20150831-20:48:26.535
56=ANOTHER_INC
22=99
48=ITRZ21
262=req_A
268=2
269=0
270=0.01005
271=10
272=20150831
273=20:48:26.514
269=1
270=0.0101
271=2
272=20150831
273=20:48:26.514
10=123

+```
+
+The FIX Message filter plugin can read the FIX log as an input and turn it into something like this:
+
+![alt tag](http://i.imgur.com/gkeStss.png)
+
+## Installation
+```
+$ /opt/logstash/bin/plugin install logstash-filter-fix_protocol
+```
+
+## Plugin Configuration
+
+| Setting                 | Input type      | Required | Default Value      |
+| ----------------------- | ----------------| ---------| ------------------ |
+| message                 | string/variable | Yes      | "Fix String"       |
+| data_dictionary_path    | string          | Yes      | "/PATH/TO/YOUR/DD" |
+| session_dictionary_path | string          | No       | nil                |
+
+**message**
++ value type is a string
++ required
+
+Should be the actual fix message passed to the filter. You might need to use a separate filter, like grok, to parse a log and set a fix string variable.
+
+**data_dictionary_path**
++ value type is a string
++ required
+
+Should be the absolute path to your data dictionary xml file.
+
+**session_dictionary_path**
++ value type is a string
++ Not required
+
+Should be the absolute path to your session dictionary xml file for FIX versions > 5.0. Note, if you do not set this but are using FIX 5.0, the filter will still work, but admin messages won't be correctly parsed - you'll lose data. The filter ignores key-value pairs that it doesn't parse correctly.
+
+**Sample Config File**
+
+*Note: For FIX < 5.0, simply omit the `session_dictionary_path`.*
+
+```
+input {
+  file {
+    path => "/PATH/TO/YOUR/FIX-MESSAGE.log"
+    start_position => "beginning"
+  }
+}
+filter {
+  grok {
+    match => ["message","%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:fix_string}: %{GREEDYDATA:fix_message}"]
+  }
+  fix_protocol {
+    message => fix_message
+    session_dictionary_path => "/PATH/TO/FIX/5.0/SESSION/DICTIONARY/FIX.xml"
+    data_dictionary_path => "/PATH/TO/FIX/5.0/DATA/DICTIONARY/FIX.xml"
+  }
+}
+output {
+  stdout { codec => rubydebug }
+}
+
+```
+
+Notice, we're using the Grok filter to create a `fix_message` variable from a theoretical FIX Message log file. Then, we're passing that variable to our filter. You can see this emulated behavior in our specs.
+
+## Development Environment
+
+To get set up quickly, we recommend using Vagrant with the Ansible provisioning available in this source repository.
+
+### Setup with Vagrant
+
+* Install [Ansible](http://www.ansible.com/)
+* Install [VirtualBox](https://www.virtualbox.org)
+* Install [Vagrant](http://www.vagrantup.com/)
+
+Then,
+
+```
+vagrant up
+```
+
+### Manual Setup (OSX)
++ `rvm install jruby`
++ `rvm use jruby`
++ `bundle install`
++ `brew install logstash`
+
+Then, run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To release a new version, update the **version number** in `logstash-filter-fix_protocol.gemspec`, and then run `bundle exec rake release` to create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+*Note: If you get an error message about metadata, you'll need to update to ruby gems > 2.0. Run `gem update --system`*
+
+### Running Tests
+
+```
+$ ./bin/rspec rspec
+```
+
+### Development Logstash Installation
+
+1. Add the filter to your installation of LogStash
+
+    ```ruby
+    # /opt/logstash/Gemfile
+    #...
+    gem "logstash-output-kafka"
+    gem "logstash-input-http_poller"
+    gem "logstash-filter-fix_protocol", :path => "/PATH/TO/YOUR/FORK"
+    ```
+
+2. Install the filter plugin
+
+    ```
+    $ /opt/logstash/bin/plugin install --no-verify
+    ```
+
+3. Start logstash installation with a LogStash configuration file.
+
+    ```
+    $ /opt/logstash/bin/logstash -f /PATH/TO/logstash.conf
+    ```
+
+## Contributing
+
+Contributions are welcome!  Please see the [Contribution Guidelines](CONTRIBUTING.md) for details.
+
+![Connamara Systems](http://www.connamara.com/wp-content/uploads/2016/01/connamara_logo_dark.png)
+
+FIX Message Logstash Filter is maintained and funded by [Connamara Systems, llc](http://connamara.com).
+
+The names and logos for Connamara Systems are trademarks of Connamara Systems, llc.
+
+## Licensing
+
+FIX Message Logstash Filter is Copyright © 2016 Connamara Systems, llc.
+
+This software is available under the Apache license and a commercial license.  Please see the [LICENSE](LICENSE.txt) file for the terms specified by the Apache license.  The commercial license offers more flexible licensing terms compared to the Apache license, and includes support services.  [Contact us](mailto:info@connamara.com) for more information on the Connamara commercial license, what it enables, and how you can start commercial development with it.
+
+This product includes software developed by quickfixengine.org ([http://www.quickfixengine.org/](http://www.quickfixengine.org/)). Please see the [QuickFIX Software LICENSE](QUICKFIX_LICENSE.txt) for the terms specified by the QuickFIX Software License.

data/lib/logstash/filters/data_dictionary.rb

@@ -0,0 +1,16 @@
+require 'quickfix'
+
+module LogStash
+  module  Filters
+    class DataDictionary < quickfix.DataDictionary
+      attr_reader :file
+
+      def initialize(file_path)
+        @file = ::File.new(file_path) # throw an exception if the file isn't found
+
+        super(file_path)
+      end
+
+    end
+  end
+end

data/lib/logstash/filters/fix_message.rb

@@ -0,0 +1,91 @@
+require 'quickfix'
+require 'active_support/core_ext'
+
+module LogStash
+  module Filters
+    class FixMessage < quickfix.Message
+      attr_reader :type, :msg_string, :session_dictionary, :data_dictionary, :all_dictionaries
+
+      def initialize(msg_string, data_dictionary, session_dictionary)
+        @session_dictionary = session_dictionary
+        @data_dictionary = data_dictionary
+        @msg_string = msg_string
+        @type = quickfix.MessageUtils.getMessageType(msg_string)
+
+        @all_dictionaries = [@data_dictionary, @session_dictionary]
+
+        super(msg_string, data_dictionary, false)
+      end
+
+      def to_hash
+        # TODO: This logic / parsing might make sense in quickfix-j / java world
+        # Then, from here, we could inherit from quickfix.Message and call `JSON.parse(self.to_json)`
+        # OR: Might want to move all this to ruby
+        # dd = Hash.from_xml(load_fixture("FIX50SP1.xml")) || https://github.com/jnunemaker/crack
+        header_msg  = field_map_to_hash(self.header)
+        body_msg    = field_map_to_hash(self, type)
+        trailer_msg = field_map_to_hash(self.trailer)
+
+        header_msg.merge(body_msg).merge(trailer_msg)
+      end
+
+      private
+
+      def field_type(tag)
+        @all_dictionaries.each do |dd|
+          enum  = dd.get_field_type_enum(tag)
+          value = enum.get_name unless enum.nil?
+          return value if value.present?
+        end
+      end
+
+      def field_name(tag)
+        @all_dictionaries.each do |dd|
+          value = dd.get_field_name(tag)
+          return value if value.present?
+        end
+      end
+
+      def field_map_to_hash(field_map, msg_type = nil)
+        hash = {}
+        # java TreeMap: https://github.com/quickfix-j/quickfixj/blob/master/quickfixj-core/src/main/java/quickfix/FieldMap.java#L395
+        iter = field_map.iterator
+
+        while iter.has_next
+          field = iter.next
+          tag   = field.get_tag
+          value = field.get_value
+
+          # IF GROUP
+          if msg_type.present? and @data_dictionary.is_group(msg_type, tag)
+            groups = []
+
+            for i in 1..value.to_i
+              group_map = field_map.get_group(i, tag)
+              groups << field_map_to_hash(group_map, msg_type)
+            end
+
+            value = groups
+          # IF FIELD
+          elsif @data_dictionary.is_field(tag)
+            value =
+              case field_type(tag)
+                when "INT", "DAYOFMONTH" then value.to_i
+                when "PRICE", "FLOAT", "QTY" then value.to_f
+                when "BOOLEAN" then value == "Y"
+                when "NUMINGROUP" then field_map.to_hash(value)
+                else
+                  value_name = @data_dictionary.get_value_name(tag, value)
+                  value_name.presence || value
+              end
+          end
+
+          hash[field_name(tag)] = value
+        end
+
+        hash
+      end
+
+    end
+  end
+end

data/lib/logstash/filters/fix_protocol.rb

@@ -0,0 +1,61 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require "logstash/filters/data_dictionary"
+require "logstash/filters/fix_message"
+
+module LogStash
+  module  Filters
+    class FixProtocol < LogStash::Filters::Base
+
+      attr_reader :data_dictionary, :session_dictionary
+
+      config_name "fix_protocol"
+
+      config :message, validate: :string, default: "Hello"
+
+      config :data_dictionary_path, validate: :string, default: "/PATH/TO/YOUR/DD"
+      config :session_dictionary_path, validate: :string, default: nil
+
+      def initialize(options = {})
+        super(options)
+
+        fail "Need to configure a valid data dictionary path" unless config["data_dictionary_path"]
+
+        @data_dictionary = DataDictionary.new(config["data_dictionary_path"])
+
+        # Set session data dictionary variable if using > FIX 5.0
+        session_dict = config["session_dictionary_path"]
+        @session_dictionary = session_dict.present? ? DataDictionary.new(session_dict) : @data_dictionary
+      end
+
+      def register
+        # just here because you complain otherwise
+      end
+
+      def filter(event)
+        if event["fix_message"]
+          fix_message = FixMessage.new(event["fix_message"], data_dictionary, session_dictionary)
+
+          fix_hash = fix_message.to_hash
+
+          fix_hash.each do |key, value|
+            begin
+              event[key] = value
+            rescue NoMethodError => e
+              puts "********"
+              puts "WARNING: Could not correctly parse #{event["fix_message"]}"
+              puts JSON.pretty_generate(fix_hash)
+              puts "Message: #{e.message}"
+              puts "********"
+            ensure
+              next
+            end
+          end
+        end
+        # filter_matched should go in the last line of our successful code
+        filter_matched(event)
+      end
+    end
+  end
+end

data/logstash-filter-fix_message_filter.gemspec

@@ -0,0 +1,34 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |s|
+  s.name          = "logstash-filter-fix_protocol"
+  s.version       = "0.1.1"
+  s.authors       = ["Connamara Systems"]
+  s.email         = ["info@connamara.com"]
+
+  s.summary       = "FIX Protocol Logstash Filter"
+  s.description   = "Put your financial application logs to work with logstash FIX filtering"
+  s.homepage      = "https://github.com/connamara/logstash-filter-fix_protocol"
+  s.licenses      = ['Apache License (2.0)']
+
+  s.files         = Dir['lib/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE.txt','QUICKFIX_LICENSE.txt','NOTICE.TXT', 'spec/**/*', 'features/**/*']
+
+  s.test_files    = s.files.grep(%r{^(spec|features)/})
+
+  s.require_paths = ["lib"]
+
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
+
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
+  s.add_runtime_dependency "logstash-input-generator"
+  s.add_runtime_dependency "activesupport"
+  s.add_runtime_dependency "quickfix-jruby"
+
+  s.add_development_dependency "logstash-devutils"
+  s.add_development_dependency "bundler", "~> 1.8"
+  s.add_development_dependency "rake", "~> 10.0"
+  s.add_development_dependency "rspec"
+  s.add_development_dependency "pry"
+end