hsbc_pdf_statement_parser 1.0.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1aeed5005a826d427a328576db61241bab262e44941ae1c0af5d34f7028f1281
-  data.tar.gz: 1af68498d1b4e4732e42a7422a32930170f4870f85a458b0a166b95e0ef7b338
+  metadata.gz: 67d78ba59d35520d3f1c12ba39f5d53358a68c6d6a1eb7fc2aa8e9180a24db40
+  data.tar.gz: 5bf2f6a30605cc92ba6c780051956a017b9ba1e4cd973e4ba6e671bece3f48f7
 SHA512:
-  metadata.gz: 64d6adeb4671eff35241d94d8f848fe54cbf6c324aa366e75d658d44b5113b468badde10b262ab092ab8b43e87402dad81cc10f9ba4b281c6fefc325c4533199
-  data.tar.gz: bebec0676834fd0da389f2094017a930c795f103a7477e67062449d31c3598cd9571f5f6dcbb57ab06549e4b761515915e9f211d936bf1e7660452c2497a32db
+  metadata.gz: 2cb49c96f8ec208224928924e9bedcfbe1287b0c1847c1ced480fa67ff3ceb50d5b643aa04885a09b627dcb48fab249b152aa579a9b3e40c050862ada3e11af1
+  data.tar.gz: 4da639fa609cebb7101e15d47e36b5891486a649e4288884177dfe92373c6d429ecd756893e3e2525b77eff8e465e6a01a12736e37d9c0344a40c3dfb4f56f96

data/README.md

@@ -0,0 +1,93 @@
+# HSBC PDF Statement Parser
+
+This is a _very_ quick and dirty gem that swallows downloaded PDF files from HSBC (UK) and parses them into a [Dry::Struct](https://dry-rb.org/gems/dry-struct/1.0/) containing details of the statement + its transactions.
+
+It exists soley because HSBC doesn’t seem to offer any way of exporting old statements as anything other than PDFs, which makes it a pain in the backside to import anything into any kind of finance packages.
+You probably shouldn’t use it (see warnings below)
+
+# Installation
+
+Using bundler on the command line:
+
+```shell
+$ bundle add hsbc_pdf_statement_parser
+$ bundle
+```
+
+## Usage
+
+This gem exposes one method: `parse`, which takes the path to a statement PDF and returns a `Dry::Struct` representation.
+
+```ruby
+require 'hsbc_pdf_statement_parser'
+
+parsed = HsbcPdfStatementParser.parse( 'path/to/statement.pdf' )
+parsed.transactions.each do |tx|
+  printf( 
+    "[%s] {%-3s} %-40s %7.02f  |  %7.02f\n", 
+    tx.date, 
+    tx.type,
+    tx.details.lines.first.strip, 
+    tx.change,
+    tx.balance
+  )
+end
+```
+
+### Statement Properties
+
+- `account_holder`: the name of the account holder _(String)_
+- `sortcode`: the sortcode shown on the statement _(String)_
+- `account_number`: the account number shown on the statement _(String)_
+- `sheets`: the sheets used in the statement _(Range[Int])_
+- `date_range`: the date range shown on the first page of the statement _(Range[Date])_
+- `opening_balance`: the opening balance of the statement _(Decimal)_
+- `closing_balance`: the closing balance of the statement _(Decimal)_
+- `payments_in`: the total of all transactions into the account _(Decimal)_
+- `payments_out`: the total of all transactions out of the account _(Decimal)_
+
+**Note:** `payments_in` and `payments_out` are those shown on the first page of the statement and they are not calculated from- or checked against the parsed transactions.
+
+Also note that sheet numbers are not guaranteed to be unique. Not sure why this is the case, but I have a few statements where the the last sheet of one statement and the first of another have the same sheet number.
+
+### Transaction Properties
+
+- `date`: the date of the transaction _(Date)_
+- `type`: a string representation of the type of the transaction (eg. `DD` for a direct debit, `VIS` for VISA, etc) _(String)_
+- `details`: a text description of the transaction, which may span multiple lines _(String)_
+- `paid_in`: the amount paid in, if appropriate _(Decimal, nullable)_
+- `paid_out`: the amount paid out, if appropriate _(Decimal, nullable)_
+- `balance`: the balance of the account after the transaction _(Decimal)_
+- `change`: the calculated change to the balance of the account: negative for debits, positive for credits _(Decimal)_
+
+**Note:** unlike in V1, `balance` is now always present and is calculated as a running total based on the opening balance of the statement. Where the statement shows a running balance after transactions (seems to be once a day), this is checked and the parser will raise an error if any discrepancy is found.
+
+## ⚠️ Warnings
+
+This gem has been thrown together for my own needs, and is offered to the world just in case someone else might want to play around with it. 
+It seems to work pretty well with statements from my Advance account here in the UK, and may also work with other flavours of accounts from elsewhere in the world, but comes with absolutely zero guarantees or assurances.
+
+That is to say: it seems to work OK for mucking around, but I’d recommend not using it for anything mission-critical, or in a situation that might lead you or others into making any kind of financial decisions.
+Any dumb financial decisions made are entirely on you =)
+
+### Wot? No tests
+
+I have plenty, sadly the only way of properly testing this code is by parsing real bank statements, and I’m not about to commit any of those to github. Sorry!
+
+## Upgrading from V1.x
+
+For various reasons I’ve not worried too much about trying to maintain backward compatibility: migration should be relatively minimal, though:
+
+### Breaking changes
+
+1. invocation: `HsbcPdfStatementParser::Parser.new(…)` becomes `HsbcPdfStatementParser.parse(…)`
+2. when using parsed transactions, `in` and `out` are now `paid_in` and `paid_out` respectively
+3. any use of `fetch(…)` on parsed transactions will need to be replaced with bare function calls (hash accessors—ie `tx[:date]`—will continue to work)
+
+### Nonbreaking changes
+
+Aside from the new properties added to the main Statement type, the biggest difference is that a statement’s `balance` property is now always specified, whereas it was only specified once a day in V1.x
+
+---
+
+Share and enjoy

data/hsbc_pdf_statement_parser.gemspec

@@ -3,18 +3,17 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require "hsbc_pdf_statement_parser/version"
 
 Gem::Specification.new do |spec|
-  
-  spec.name     = 'hsbc_pdf_statement_parser'
-  spec.summary  = 'Quick and dirty RubyGem to parse HSBC’s statement PDFs'
-  spec.license  = 'MIT'
-  
-  spec.authors  = 'Jon Pearse'
-  spec.email    = 'hello@jonpearse.net'
-  spec.homepage = 'https://github.com/jonpearse/hsbc-pdf-statement-parser'
-  
-  spec.version  = HsbcPdfStatementParser::VERSION
-  spec.files    = `git ls-files`.split($\)
-  
-  spec.add_dependency 'pdf-reader', '~> 2.4'
-  
+  spec.name = "hsbc_pdf_statement_parser"
+  spec.summary = "Quick and dirty RubyGem to parse HSBC’s statement PDFs"
+  spec.license = "MIT"
+
+  spec.authors = "Jon Pearse"
+  spec.email = "hello@jonpearse.net"
+  spec.homepage = "https://github.com/jonpearse/hsbc-pdf-statement-parser"
+
+  spec.version = HsbcPdfStatementParser::VERSION
+  spec.files = `git ls-files`.split($\)
+
+  spec.add_dependency "pdf-reader", "~> 2.9"
+  spec.add_dependency "dry-struct", "~> 1.4"
 end

data/lib/hsbc_pdf_statement_parser/reader.rb

@@ -0,0 +1,24 @@
+module HsbcPdfStatementParser
+  class Reader
+    def initialize(filename)
+      @reader = PDF::Reader.new(filename)
+    end
+
+    def first_page
+      @_first_page ||= @reader.pages.first.text
+    end
+
+    def all_text
+      @_all_text ||= @reader.pages.map(&:text).join
+    end
+
+    def statement_blocks
+      @_statement_lines ||= begin
+          @reader.pages.map do |page|
+            match = page.text.match(/BALANCE\s?BROUGHT\s?FORWARD(?:.*?)\n(.*?)BALANCE\s?CARRIED\s?FORWARD/im)
+            match ? match[1] : nil
+          end.compact
+        end
+    end
+  end
+end

data/lib/hsbc_pdf_statement_parser/statement_parser.rb

@@ -0,0 +1,125 @@
+module HsbcPdfStatementParser
+  class StatementParser
+    attr_reader :_statement_lines
+
+    def initialize(filename)
+      @reader = Reader.new(filename)
+
+      # puts @reader.first_page
+    end
+
+    def parse
+      opening_balance = scan_figure("Opening Balance")
+      closing_balance = scan_figure("Closing Balance")
+      meta = get_meta
+
+      ImportedStatement.new(
+        account_holder: meta[:account_holder],
+        sortcode: meta[:sortcode],
+        account_number: meta[:account_no],
+        sheets: meta[:sheets],
+        date_range: get_date_range,
+        opening_balance: opening_balance,
+        closing_balance: closing_balance,
+        payments_in: scan_figure("Payments In"),
+        payments_out: scan_figure("Payments Out"),
+        transactions: parse_transactions(opening_balance),
+      )
+    end
+
+    private def scan_figure(search_string)
+      # note: I do not like how … general this regex is, but HSBC seems to love putting random noise characters into
+      # opening and closing balance, so there’s only so much I can do about it.
+      match = Regexp.new("#{search_string}(.*?)\n", Regexp::IGNORECASE).match(@reader.first_page)
+      raise ParsingError.new("Could not find #{search_string}") if match.nil?
+
+      match[1].strip.gsub(",", "").gsub(/\s/, "").to_d
+    end
+
+    private def get_meta
+      all_meta = @reader.all_text.scan(/Account\s?Name\s+Sortcode\s+Account\s?Number\s+Sheet Number\n+([A-Z\s]+?)\s\s+([\d\-]+)\s\s+(\d+)\s\s+(\d+)\n/i)
+      raise ParsingError.new("Cannot find statement metadata") if all_meta.empty?
+
+      # check everything makes sense
+      raise ParsingError.new("Error parsing account name") unless all_elements_same(all_meta.map(&:first))
+      raise ParsingError.new("Error parsing sort code") unless all_elements_same(all_meta.map { |a| a[1] })
+      raise ParsingError.new("Error parsing account number") unless all_elements_same(all_meta.map { |a| a[2] })
+
+      # get page numbers
+      first_page = all_meta.first[3]
+      last_page = all_meta.last[3]
+      raise ParsingError.new("Error parsing sheet numbers") if (first_page > last_page)
+
+      {
+        sheets: Range.new(first_page.to_i, last_page.to_i),
+        account_holder: all_meta.dig(0, 0).strip,
+        sortcode: all_meta.dig(0, 1).strip,
+        account_no: all_meta.dig(0, 2).strip,
+      }
+    end
+
+    private def get_date_range
+      dates = @reader.first_page.match(/(\d{1,2}) ([a-z]+)(?: (\d{4}))? to (\d{1,2}) ([a-z]+) (\d{4})/i)
+      raise ParsingError.new("Cannot find date range") unless dates
+      dates = %i{start_day start_month start_year end_day end_month end_year}.zip(dates.captures).to_h
+
+      # default the start year
+      dates[:start_year] ||= dates[:end_year]
+      start_date = Date.parse("#{dates[:start_day]} #{dates[:start_month]} #{dates[:start_year]}")
+      end_date = Date.parse("#{dates[:end_day]} #{dates[:end_month]} #{dates[:end_year]}")
+      raise ParsingError.new("Error parsing date range") if (start_date > end_date)
+
+      Range.new(start_date, end_date)
+    end
+
+    private def parse_transactions(opening_balance)
+      # Get the raw information out of the PDF text
+      parser = TransactionParser.new
+      transactions = parser.parse(@reader.statement_blocks)
+
+      # start crosschecking!
+      running_balance = opening_balance
+      transactions.map do |tx|
+
+        # work out what we’re doing
+        running_balance += tx[:paid_in] || 0 - tx[:paid_out]
+
+        # push a calculated balance in if none is present
+        if tx[:balance].nil?
+          tx[:balance] = running_balance
+        else
+          raise TransactionCalculationError.new(tx[:balance], running_balance) if running_balance != tx[:balance]
+        end
+
+        # all good, so create a new Transaction object
+        Transaction.new(tx)
+      end
+    end
+
+    private def all_elements_same(arry)
+      arry.uniq.length == 1
+    end
+  end
+
+  class ParsingError < StandardError
+    attr_reader :info
+
+    def initialize(message, info = nil)
+      @info = info
+      super(message)
+    end
+  end
+
+  class TransactionCalculationError < StandardError
+    attr_reader :expected, :calculated
+
+    def initialize(expected, calculated)
+      @expected = expected
+      @calculated = calculated
+    end
+
+    def message
+      "Expected #{@expected} but got #{@calculated}"
+    end
+  end
+end

data/lib/hsbc_pdf_statement_parser/transaction_parser.rb

@@ -0,0 +1,133 @@
+require "bigdecimal/util"
+require "date"
+
+module HsbcPdfStatementParser
+  class TransactionParser
+    attr_reader :transactions
+
+    def parse(input)
+      # We need to parse pages individually because the column widths may not be the same on each page
+      parsed_lines = input.reduce([]) { |lines, page| lines + parse_page(page) }
+
+      # build out transactions from our parsed data
+      @transactions = parse_transactions(parsed_lines)
+    end
+
+    private def parse_page(page_data)
+      # get the maximum line length for this page
+      max_line_length = page_data.lines.map(&:length).max
+
+      # work out the columns being used
+      column_indices = find_columns(page_data, max_line_length)
+
+      # get the column keys and produce a map for parsing
+      column_keys = get_column_keys(column_indices, max_line_length)
+      column_map = column_keys.zip(column_indices).to_h
+
+      # parse out all the
+      page_data.lines.reduce([]) do |parsed, line|
+        next parsed if line.strip.empty?
+
+        parsed << parse_line(line, column_map, max_line_length)
+      end
+    end
+
+    private def find_columns(input, max_line_length)
+      state = Array.new(max_line_length, " ")
+
+      # scan the content + look for obvious columns
+      input.lines.each do |line|
+        # bounce if there’s no point…
+        next if line.strip.empty?
+
+        # map the state
+        line.split("").each.with_index { |chr, i| state[i] = "X" if chr != " " }
+      end
+
+      # We will almost have something like XX XXX XX for the date column, so replace any spaces with an X either side
+      # with an X
+      state = state.join.gsub(/X X/, "XXX")
+
+      # find the columns
+      columns = []
+      state.gsub(/X+/) { columns << Range.new(*$~.offset(0)) }
+      columns
+    end
+
+    private def get_column_keys(indices, max_line_length)
+      # if we have 6 columns, we’re all good…
+      return %i{date type details paid_out paid_in balance} if indices.length == 6
+
+      # if a statement contains no payments in or no payments out, it’ll only have 5 columns, so we need to work out
+      # which the additional column is
+      # We can do this with an assumption that the ‘paid out’ column is usually at around 2/3 of the total line length…
+      has_paid_out = indices.map { |i| i.first.to_f / max_line_length }.any? { |p| p > 0.6 && p < 0.7 }
+      additional = has_paid_out ? :paid_out : :paid_in
+
+      [:date, :type, :details, additional, :balance]
+    end
+
+    private def parse_line(line, map, max_line_length)
+      # lengthen the line
+      line = line.rstrip.ljust(max_line_length)
+
+      # cut things up
+      row = {
+        paid_out: nil,
+        paid_in: nil,
+      }
+      map.each { |k, v| row[k] = empty_to_nil(line[v]) }
+
+      # further processing
+      row[:date] = Date.strptime(row[:date], "%d %b %y") unless row[:date].nil?
+      row[:paid_out] = parse_decimal(row[:paid_out])
+      row[:paid_in] = parse_decimal(row[:paid_in])
+      row[:balance] = parse_decimal(row[:balance])
+
+      row
+    end
+
+    private def parse_decimal(str)
+      return nil if str.nil?
+
+      mult = str.end_with?("D") ? -1 : 1
+
+      str.gsub(",", "").to_d * mult
+    end
+
+    private def empty_to_nil(str)
+      str = str.strip
+
+      str.empty? ? nil : str
+    end
+
+    private def parse_transactions(parsed_lines)
+      current_date = nil
+      current_transaction = nil
+
+      transactions = []
+
+      parsed_lines.each do |parsed|
+        # if we have a new date, store it
+        current_date = parsed[:date] unless parsed[:date].nil?
+
+        # if we have a type, it’s a new transaction, so push any existing info onto the list + reset
+        unless parsed[:type].nil?
+          transactions << current_transaction unless current_transaction.nil?
+          current_transaction = parsed.merge(date: current_date)
+          next
+        end
+
+        # otherwise, merge things together
+        current_transaction = current_transaction.merge(
+          parsed.reject { |_, v| v.nil? },
+          { details: "#{current_transaction[:details]}\n#{parsed[:details]}".strip }
+        )
+      end
+
+      # shove the last transaction onto the list + return
+      transactions << current_transaction
+      transactions
+    end
+  end
+end

data/lib/hsbc_pdf_statement_parser/version.rb

@@ -1,3 +1,3 @@
 module HsbcPdfStatementParser
-  VERSION = "1.0.1"
+  VERSION = "2.0.0"
 end

data/lib/hsbc_pdf_statement_parser.rb

@@ -1,7 +1,52 @@
-require 'date'
-require 'pdf-reader'
-require 'hsbc_pdf_statement_parser/pdf_reader_patch'
-require 'hsbc_pdf_statement_parser/parser'
+require "bigdecimal/util"
+require "date"
+require "dry-struct"
+require "pdf-reader"
+
+require "hsbc_pdf_statement_parser/reader"
+require "hsbc_pdf_statement_parser/statement_parser"
+require "hsbc_pdf_statement_parser/transaction_parser"
 
 module HsbcPdfStatementParser
-end
+  module Types
+    include Dry.Types()
+  end
+
+  class Transaction < Dry::Struct
+    attribute :date, Types::Date
+    attribute :type, Types::String
+    attribute :details, Types::String
+    attribute :paid_out, Types::Decimal.optional
+    attribute :paid_in, Types::Decimal.optional
+    attribute :balance, Types::Decimal
+
+    def change
+      paid_in || 0 - paid_out
+    end
+  end
+
+  class ImportedStatement < Dry::Struct
+    attribute :account_holder, Types::String
+    attribute :sortcode, Types::String
+    attribute :account_number, Types::String
+
+    attribute :sheets, Types::Instance(Range)
+    attribute :date_range, Types::Instance(Range)
+
+    attribute :opening_balance, Types::Decimal
+    attribute :closing_balance, Types::Decimal
+    attribute :payments_in, Types::Decimal
+    attribute :payments_out, Types::Decimal
+
+    attribute :transactions, Types::Array.of(Transaction)
+  end
+
+  # Parses the passed PDF file and returns an ImportedStatement
+  #
+  # === Parameters
+  #
+  # [filename] the filename to parse
+  def self.parse(filename)
+    StatementParser.new(filename).parse
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hsbc_pdf_statement_parser
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 2.0.0
 platform: ruby
 authors:
 - Jon Pearse
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-11-26 00:00:00.000000000 Z
+date: 2022-02-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: pdf-reader
@@ -16,15 +16,29 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.4'
+        version: '2.9'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.4'
-description: 
+        version: '2.9'
+- !ruby/object:Gem::Dependency
+  name: dry-struct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+description:
 email: hello@jonpearse.net
 executables: []
 extensions: []
@@ -32,17 +46,18 @@ extra_rdoc_files: []
 files:
 - Gemfile
 - LICENSE.txt
-- README.textile
+- README.md
 - hsbc_pdf_statement_parser.gemspec
 - lib/hsbc_pdf_statement_parser.rb
-- lib/hsbc_pdf_statement_parser/parser.rb
-- lib/hsbc_pdf_statement_parser/pdf_reader_patch.rb
+- lib/hsbc_pdf_statement_parser/reader.rb
+- lib/hsbc_pdf_statement_parser/statement_parser.rb
+- lib/hsbc_pdf_statement_parser/transaction_parser.rb
 - lib/hsbc_pdf_statement_parser/version.rb
 homepage: https://github.com/jonpearse/hsbc-pdf-statement-parser
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -57,8 +72,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
-signing_key: 
+rubygems_version: 3.2.22
+signing_key:
 specification_version: 4
 summary: Quick and dirty RubyGem to parse HSBC’s statement PDFs
 test_files: []

data/README.textile

@@ -1,62 +0,0 @@
-h1. HSBC PDF Statement Parser
-
-This is a _very_ quick and dirty gem that swallows downloaded PDF files from HSBC (UK) and parses them into an array of hashes containing each transaction.
-
-It exists soley because HSBC doesn’t seem to offer any way of exporting old statements as anything other than PDFs, which makes it a pain in the backside to import anything into any kind of finance packages.
-You probably shouldn’t use it (see warnings below)
-
-h2. Installation
-
-Using bundler on the command line:
-
-<pre>
-$ bundle add hsbc_pdf_statement_parser
-$ bundle
-</pre>
-
-h2. Usage
-
-<pre>
-require 'hsbc_pdf_statement_parser'
-
-parser = HsbcPdfStatementParser::Parser.new( 'path/to/statement.pdf' )
-parser.transactions.each do |tx|
-  
-  printf( "%s: %-40s %7.02f\n", tx[:date], tx[:details].lines.first.strip, tx[:change] )
-
-end
-</pre>
-
-h3. Methods
-
-- @statements@ := returns transactions as an array of hashes (see hash keys section, below) _(Array Hash)_
-- @opening_balance@ := returns the opening balance of the statement _(Float)_
-- @closing_balance@ := returns the closing balance of the statement _(Float)_
-- @payments_in@ := returns the total of all payments in _(Float)_
-- @payments_out@ := returns the total of all payments out _(Float)_
-
-Note that @payments_in@ and @payments_out@ are read from the header section at the top of the statement, and are not calculated from the transactions in the statement.
-
-h3. Transaction hash keys
-
-- @:date@ := a `Date` object describing the date of the transaction _(Date)_
-- @:type@ := the type of the description, eg `DD` for a direct debit, `VIS` for visa, `(((` for contactless _(String)_
-- @:details@ := the text descrption of the transaction. This may span multiple lines _(String)_
-- @:in@ := the amount entering your account, or nil if an outbound transaction _(Float, nil)_
-- @:out@ := the amount leaving your account, or nil if an inbound transaction _(Float, nil)_
-- @:change@ := a calculated field showing the change to your bank balance: negative for debits, positive for credits _(Float)_
-- @:balance@ := the balance of your account after the transaction, if present in the PDF _(Float, nil)_
-
-*Note:* that the @:balance@ key is pulled straight from the PDF and will only be present for the last transaction on a particular day. I’m not doing anything even remotely clever here :)
-
-h2. ⚠️ Warnings
-
-This gem has been thrown together for my own needs, and is offered to the world just in case someone else might want to play around with it. 
-It seems to work pretty well with statements from my Advance account here in the UK, and may also work with other flavours of accounts from elsewhere in the world, but comes with absolutely zero guarantees or assurances.
-
-That is to say: it seems to work OK for mucking around, but I’d recommend not using it for anything mission-critical, or in a situation that might lead you or others into making any kind of financial decisions.
-Any dumb financial decisions made are entirely on you =)
-
-Also, this gem contains a patch for "pdf-reader":https://github.com/yob/pdf-reader to help it better cope with the weird way in which HSBC seems to generate PDFs. This is unapologetically a massive hack, and really could do with someone far smarter than me to come up with a better solution.
-For more information, see "pdf-reader issue #169":https://github.com/yob/pdf-reader/issues/169 which goes into a little more detail about what’s going on (my files seem to terminate the image data with @0xE0@, per my patch)
-

data/lib/hsbc_pdf_statement_parser/parser.rb

@@ -1,166 +0,0 @@
-module HsbcPdfStatementParser
-  
-  class Parser 
-    
-    # Creates a new parser from a PDF file.
-    #
-    # === Parameters
-    #
-    # [filename] the filename to parse
-    def initialize( filename )
-      
-      @reader = PDF::Reader.new( filename )
-      
-    end
-    
-    # Returns an array of the transactions in the document as hashes.
-    #
-    # === Hash keys
-    #
-    # [:date] the date of the transaction _(Date)_
-    # [:type] the type of the transaction, eg ‘VISA’, ‘DD’, ‘ATM’, etc _(String)_
-    # [:details] the details of the transaction. This can span multiple lines _(String)_
-    # [:out] the amount of the transaction, if a debit _(Float, nil)_
-    # [:in] the amount of the transaction, if a credit _(Float, nil)_
-    # [:change] the amount of the transacation: negative if a debit, positive if a credit _(Float)_
-    def transactions
-      
-      @_transactions ||= begin
-        
-        current_transaction = nil
-        current_date = nil
-        transactions = []
-        
-        document_text
-          .scan( /BALANCE\s?BROUGHT\s?FORWARD(?:.*?)\n(.*?)BALANCE\s?CARRIED\s?FORWARD/im )
-          .map{ |text| parse_page( text[0] )}
-          .flatten
-          .each do |line|
-          
-            # store the current date
-            current_date = line[:date] unless line[:date].nil?
-          
-            # if we have a type, start a new transaction
-            unless line[:type].nil?
-              transactions << current_transaction unless current_transaction.nil?
-              current_transaction = line.merge( date: current_date )
-              next
-            end
-          
-            # merge things in
-            current_transaction.merge!( line.select{ |k,v| v }, { details: "#{current_transaction[:details]}\n#{line[:details]}" })
-          
-          end
-        
-        # dump the final transaction + return
-        transactions << current_transaction unless current_transaction.nil?
-        transactions
-      end
-      
-    end
-    
-    # Returns the opening balance of the statement read from the table on the first page.
-    def opening_balance
-      
-      @_opening_balance ||= scan_figure( 'Opening Balance' )
-      
-    end
-    
-    # Returns the closing balance of the statement read from the table on the first page.
-    def closing_balance
-      
-      @_closing_balance ||= scan_figure( 'Closing Balance' )
-      
-    end
-    
-    # Returns the total value of payments in during the statement read from the table on the first page (ie: not calculated)
-    def payments_in
-      
-      @_payments_in ||= scan_figure( 'Payments In' )
-      
-    end
-    
-    # Returns the total value of payments out during the statement read from the table on the first page (ie: not calculated)
-    def payments_out
-      
-      @_payments_out ||= scan_figure( 'Payments Out' )
-      
-    end
-    
-    private
-    
-    def document_text
-      
-      @text ||= begin
-        
-        @reader.pages.map( &:text ).join
-        
-      end
-      
-    end
-    
-    def scan_figure( search_string )
-      
-      @_first_page ||= @reader.pages.first.text
-      
-      match = Regexp.new( "#{search_string}(?:.*?)([0-9\.\,]{4,})", Regexp::IGNORECASE ).match( @_first_page )
-      return nil if match.nil?
-      
-      match[1].gsub( ',', '' ).to_f
-      
-    end
-    
-    def parse_page( page_str )
-      
-      # grab lines + get the longest
-      lines = page_str.lines
-      max_length = lines.map( &:length ).max
-      
-      lines.map{ |line| parse_line( line.rstrip.ljust( max_length ))}.compact
-        
-    end
-    
-    def parse_line( line_str )
-      
-      # if we’re blank…
-      return nil if line_str.strip.empty?
-      
-      # start cutting things up
-      row = {
-        date:    empty_to_nil( line_str[0..12] ),
-        type:    empty_to_nil( line_str[12..20] ),
-        details: empty_to_nil( line_str[20..70] ),
-        out:     empty_to_nil( line_str[70..90] ),
-        in:      empty_to_nil( line_str[90..110] ),
-        balance: empty_to_nil( line_str[110..130] )
-      }
-      
-      # munge things further
-      row[:date]    = Date.strptime( row[:date], '%d %b %y' ) unless row[:date].nil?
-      row[:out]     = row[:out].gsub( ',', '' ).to_f unless row[:out].nil?
-      row[:in]      = row[:in].gsub( ',', '' ).to_f unless row[:in].nil?
-      row[:balance] = row[:balance].gsub( ',', '' ).to_f unless row[:balance].nil?
-      
-      # set a change amount
-      row[:change]  = if !row[:out].nil?
-        0 - row[:out]
-      elsif !row[:in].nil?
-        row[:in]
-      else
-        nil
-      end
-      
-      # return
-      row
-        
-    end
-    
-    def empty_to_nil( str )
-      
-      ( str.strip!.empty? ) ? nil : str
-      
-    end
-    
-  end
-  
-end

data/lib/hsbc_pdf_statement_parser/pdf_reader_patch.rb

@@ -1,44 +0,0 @@
-# This is a horrendous patch to work around issue #169 in the pdf-reader repo (https://github.com/yob/pdf-reader/issues/169).
-#
-# Short version is that whatever HSBC is using to generate PDFs doesn’t seem to null-/whitespace-terminate inline image
-# data. Thus, when PdfReader tries to find the ‘EI’ token when parsing inline media, it can’t and simply runs off the end
-# of the document, causing a TypeError to be thrown.
-#
-# The PDF files I’m getting all seem to end some of the images with xE0, so I’ve simply monkey-patched this into the
-# library for use with my files.
-# This may not be the case for anyone else, in which case maybe add whatever your problem character is to the regex and
-# open a PR should you feel the need.
-# 
-# Or, y’know, look at the PdfReader source and see if you can work out something better, because this is horrendous :/
-
-module PdfReaderPatch
-  def self.included( base )
-    base.class_eval do      
-      def prepare_inline_token
-        str = "".dup
-
-        buffer = []
-        to_rewind = -3
-
-        until buffer[0] =~ /\s|\0|\xE0/n && buffer[1, 2] == ['E', 'I']
-          chr = @io.read(1)
-          buffer << chr
-          
-          if buffer.length > 3
-            str << buffer.shift
-          end
-          
-          to_rewind = -2 if buffer.first =~ /\xE0/n
-        end
-
-        str << '\0' if buffer.first == '\0'
-
-        @tokens << string_token(str)
-              
-        @io.seek(to_rewind, IO::SEEK_CUR) unless chr.nil?      
-      end
-    end
-  end
-end
-
-PDF::Reader::Buffer.send( :include, PdfReaderPatch )