cmxl 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 471c7b0121f9cd95bafcfe16586d35b9cd9ec50a
+  data.tar.gz: 5694a0e0d8089bc656aae5baa3c923277e7ad8ed
+SHA512:
+  metadata.gz: 9a58700e14cda9fca9b787e70970630324800606f4e5652f588bcb1fa5f9d9f9f8e7f91269d98e7d2d5be67f002be64bb2fc54cfb06f47151f71df0640185157
+  data.tar.gz: a3e652b2dec634c5b1bfe9292a65ae83f72a18a0402ca7d3c8ca8b253d4b730f0ceb2367a22779f0968769e6720cb744e96de4a6a9f7147064da35b7fda639ae

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in cmxl.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Michael Bumann
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,136 @@
+# Cmxl - your friendly ruby MT940 parser
+
+At Railslove we build a lot of banking and payment applications and work on integrating applications with banks and banking functionality. 
+Our goal is to making it easy with what sometimes seems complicated. 
+
+Cmxl is a friendly and extendible MT940 bank statement file parser that helps your extracting data from the bank statement files. 
+
+## What is MT940?
+
+MT940 (MT = Message Type) is the SWIFT-Standard for the electronic transfer of bank statement files. 
+When integrating with banks you often get MT940 files as interface. 
+For more information have a look at the different [SWIFT message types](http://en.wikipedia.org/wiki/SWIFT_message_types)
+
+At some point in the future MT940 file should be exchanged with newer XML documents - but banking institutions are slow so MT940 will stick around for a while.
+
+## Reqirements
+
+Cmxl is a pure ruby parser and has no gem dependencies. 
+
+* Ruby 1.9.3 or newer
+
+## Installation
+
+
+Add this line to your application's Gemfile:
+
+    gem 'cmxl'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install cmxl
+
+## Usage
+
+Simple usage: 
+
+```ruby
+
+statements = Cmxl.parse(File.read('mt940.txt')) # parses the file and returns an array of statement objects
+statements.each do |s|
+  puts s.reference
+  puts s.generation_date
+  puts s.opening_balance.amount
+  puts s.closing_balance.amount
+  puts.sha # SHA of the statement source - could be used as an unique identifier
+  
+  s.transactions.each do |t|
+    puts t.information
+    puts t.description
+    puts t.entry_date
+    puts t.funds_code
+    puts t.credit?
+    puts t.debit?
+    puts t.sign # -1 if it's a debit; 1 if it's a credit
+    puts t.name
+    puts t.iban
+    puts t.sepa
+    puts t.sub_fields
+    puts t.reference
+    puts t.bank_reference
+    # ...
+  end
+end
+
+```
+
+Every object responds to `to_h` and let's you easily convert the data to a hash. 
+
+#### A note about encoding and file wirednesses
+
+You probably will encounter encoding issues (hey, you are building banking applications!). 
+We try to handle encoding and format wirednesses as much as possible.  
+If you have encoding issues you can pass encoding options to the `Cmxl.parse(<string>, <options hash>)` it accepts the same options as [String#encode](http://ruby-doc.org/core-2.1.3/String.html#method-i-encode)
+If that fails try to motify the file before you pass it to the parser - and please create an issue.
+
+### Custom field parsers
+
+Because a lot of banks implement the MT940 format slightly different one of the design goals of this library is to be able to customize the field parsers. 
+Every line get parsed with a special parser. Here is how to write your own parser:
+
+```ruby
+
+# simply create a new parser class inheriting from Cmxl::Field
+class MyFieldParser < Cmxl::Field
+  self.tag = 42 # define which MT940 tag your parser can handle. This will automatically register your parser and overwriting existing parsers
+  self.parser = /(?<world>.*)/ # the regex to parse the line. Use named regexp to access your match.
+
+  def upcased
+    self.data['world'].upcase
+  end
+end
+
+my_field_parser = MyFieldParser.parse(":42:hello from mt940")
+my_field_parser.world #=> hello from MT940
+my_field_parser.upcased #=> HELLO FROM MT940
+my_field_parser.data #=> {'world' => 'hello from mt940'} - data is the accessor to the regexp matches
+
+```
+
+## Parsing issues? - please create an issue with your file
+
+The Mt940 format often looks different for the different banks and the different countries. Especially the not strict defined fields are often used for custom bank data. 
+If you have a file that can not be parsed please open an issue. We hope to build a parser that handles most of the files.
+
+
+## ToDo
+
+* collect MT940 files from different banks and use them as example for specs
+
+
+## Contributing
+
+### Specs
+We use rspec to test Cmxl. Run `rake` to execute the whole test suite.
+
+1. Fork it ( http://github.com/railslove/cmxl/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## Credits and other parsers
+
+Cmxl is inspired and borrows ideas from the `mt940_parser` by the great people at [betterplace](https://www.betterplace.org/).
+
+other parsers:
+* [betterplace/mt940_parser](https://github.com/betterplace/mt940_parser)
+* [gmitrev/mt940parser](https://github.com/gmitrev/mt940parser)
+
+------------
+
+2014 - built with love by [Railslove](http://railslove.com). We have built quite a number of FinTech products. If you need support we are happy to help. Please contact us at team@railslove.com.

data/Rakefile

@@ -0,0 +1,7 @@
+require "bundler/gem_tasks"
+
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/cmxl.gemspec

@@ -0,0 +1,24 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'cmxl/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "cmxl"
+  spec.version       = Cmxl::VERSION
+  spec.authors       = ["Michael Bumann"]
+  spec.email         = ["michael@railslove.com"]
+  spec.summary       = %q{Cmxl is your friendly MT940 bank statement parser}
+  spec.description   = %q{Cmxl provides an extensible and customizable parser for the MT940 bank statement format.}
+  spec.homepage      = "http://railslove.com"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.5"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec", '~>3.0'
+end

data/lib/cmxl.rb

@@ -0,0 +1,20 @@
+require "cmxl/version"
+
+require 'cmxl/field'
+require 'cmxl/statement'
+require 'cmxl/transaction'
+Dir[File.join(File.dirname(__FILE__), 'cmxl/fields', '*.rb')].each { |f| require f; }
+module Cmxl
+
+  STATEMENT_SEPARATOR = "\n-"
+  def self.parse(data, options={})
+    options[:universal_newline] ||= true
+    if options[:encoding]
+      data.encode!('UTF-8', options.delete(:encoding), options)
+    else
+      data.encode!('UTF-8', options) if !options.empty?
+    end
+
+    data.split(STATEMENT_SEPARATOR).reject { |s| s.strip.empty? }.collect {|s| Cmxl::Statement.new(s.strip) }
+  end
+end

data/lib/cmxl/field.rb

@@ -0,0 +1,78 @@
+require 'date'
+module Cmxl
+  class Field
+    class LineFormatError < StandardError; end
+    class Unknown < Field;
+      @parser= /(?<source>.*)/
+      def to_h
+        { tag: tag, modifier: modifier, source: source }
+      end
+    end
+    DATE = /(?<year>\d{0,2})(?<month>\d{2})(?<day>\d{2})/
+    attr_accessor :source, :modifier, :match, :data, :tag
+
+    @@parsers = {}
+    @@parsers.default = Unknown
+    def self.parsers; @@parsers; end
+
+    class << self; attr_accessor :parser; end
+    def self.tag=(tag)
+      @tag = tag.to_s
+      @@parsers[tag.to_s] = self
+    end
+    def self.tag
+      @tag
+    end
+
+    def self.parse(line)
+      if line.match(/^:(\d{2,2})(\w)?:(.*)$/)
+        tag, modifier, content = $1, $2, $3
+        Field.parsers[tag.to_s].new(content, modifier, tag)
+      else
+        raise LineFormatError, "Wrong line format: #{line.dump}"
+      end
+    end
+
+    def initialize(source, modifier=nil, tag=nil)
+      self.tag = tag
+      self.modifier = modifier
+      self.source = source
+      self.data = {}
+
+      if self.match = self.source.match(self.class.parser)
+        self.match.names.each do |name|
+          self.data[name] = self.match[name]
+        end
+      end
+    end
+
+    def to_h
+      self.data
+    end
+    alias :to_hash :to_h
+    def to_json(*args)
+      to_h.to_json(*args)
+    end
+
+    def to_date(date, year=nil)
+      if match = date.match(DATE)
+        year ||= "20#{match['year']}"
+        month = match['month']
+        day = match['day']
+        Date.new(year.to_i, month.to_i, day.to_i)
+      else
+        date
+      end
+    rescue ArgumentError # let's simply ignore invalid dates
+      date
+    end
+
+    def method_missing(m, *value)
+      if m =~ /=\z/
+        self.data[m] = value.first
+      else
+        self.data[m.to_s]
+      end
+    end
+  end
+end

data/lib/cmxl/fields/account_balance.rb

@@ -0,0 +1,45 @@
+module Cmxl
+  module Fields
+    class AccountBalance < Field
+      self.tag = 60
+      self.parser = /(?<funds_code>\A[a-zA-Z]{1})(?<date>\d{6})(?<currency>[a-zA-Z]{3})(?<amount>[\d|,|\.]{4,15})/i
+
+      def date
+        to_date(self.data['date'])
+      end
+
+      def credit?
+        self.data['funds_code'].to_s.upcase == 'C'
+      end
+
+      def debit?
+        !credit?
+      end
+
+      def amount
+        self.data['amount'].gsub(',','.').to_f * self.sign
+      end
+
+      def sign
+        self.credit? ? 1 : -1
+      end
+
+      def amount_in_cents
+        self.data['amount'].gsub(',', '').gsub('.','').to_i * self.sign
+      end
+
+      def to_h
+        {
+          'date' => date,
+          'funds_code' => funds_code,
+          'credit' => credit?,
+          'debit' => debit?,
+          'currency' => currency,
+          'amount' => amount,
+          'amount_in_cents' => amount_in_cents,
+          'sign' => sign
+        }
+      end
+    end
+  end
+end

data/lib/cmxl/fields/account_identification.rb

@@ -0,0 +1,12 @@
+module Cmxl
+  module Fields
+    class AccountIdentification < Field
+      self.tag = 25
+      self.parser = /(?<bank_code>\w{8,11})\/(?<account_number>\d{0,23})(?<currency>[A-Z]{3})?|(?<country>[a-zA-Z]{2})(?<ban>\d{11,36})/i
+
+      def iban
+        "#{self.country}#{self.ban}"
+      end
+    end
+  end
+end

data/lib/cmxl/fields/available_balance.rb

@@ -0,0 +1,8 @@
+module Cmxl
+  module Fields
+    class AvailableBalance < AccountBalance
+      self.tag = 64
+      self.parser = superclass.parser
+    end
+  end
+end

data/lib/cmxl/fields/closing_balance.rb

@@ -0,0 +1,8 @@
+module Cmxl
+  module Fields
+    class ClosingBalance < AccountBalance
+      self.tag = 62
+      self.parser = superclass.parser
+    end
+  end
+end

data/lib/cmxl/fields/reference.rb

@@ -0,0 +1,21 @@
+module Cmxl
+  module Fields
+    class Reference < Field
+      self.tag = 20
+      self.parser = /(?<statement_identifier>[a-zA-Z]{0,2})(?<date>\d{6})(?<additional_number>.*)/i
+
+      def reference
+        self.source
+      end
+
+      def date
+        to_date(self.data['date'])
+      end
+
+      def to_h
+        super.merge({'date' => date, 'reference' => source})
+      end
+
+    end
+  end
+end

data/lib/cmxl/fields/statement_details.rb

@@ -0,0 +1,54 @@
+module Cmxl
+  module Fields
+    class StatementDetails < Field
+      self.tag = 86
+      self.parser = /(?<transaction_code>\w{3})(?<details>(?<seperator>.).*)/
+
+      def sub_fields
+        @sub_fields ||= self.data['details'].split(/#{Regexp.escape(self.data['seperator'])}(\d{2})/).reject(&:empty?).each_slice(2).to_h
+      end
+
+      def description
+        self.sub_fields['00']
+      end
+
+      def information
+        (20..29).to_a.collect {|i| self.sub_fields[i.to_s] }.join('')
+      end
+
+      def sepa
+        if self.information =~ /([A-Z]{4})\+/
+          self.information.split(/([A-Z]{4})\+/).reject(&:empty?).each_slice(2).to_h
+        else
+          {}
+        end
+      end
+
+      def bic
+        self.sub_fields['30']
+      end
+
+      def name
+        "#{self.sub_fields['32']}#{self.sub_fields['33']}"
+      end
+
+      def iban
+        self.sub_fields['38'] || self.sub_fields['31']
+      end
+
+      def to_h
+        {
+          'bic' => bic,
+          'iban' => iban,
+          'name' => name,
+          'sepa' => sepa,
+          'information' => information,
+          'description' => description,
+          'sub_fields' => sub_fields,
+          'transaction_code' => transaction_code,
+          'details' => details
+        }
+      end
+    end
+  end
+end