lloydstsb 0.1.0

data/.gitignore

@@ -0,0 +1 @@
+settings.rb

data/README.md

@@ -0,0 +1,58 @@
+## Lloyds TSB screen scraper
+
+I bank with Lloyds TSB - I have my current account and credit card with them. Like most online banking services though, they're not to up-to-date on APIs and the like. After looking around online, I found that there were a couple of scripts that people had built, but I didn't have much luck with them myself. So I decided to build my own screen scraper.
+
+I know the code in this is pretty messy, and as ever, it's untested. I tried to refactor it and got to the end, but then it turned out to be broken and I couldn't be bothered to fix it. So I've left it for now.
+
+### Usage
+
+The file `example.rb` provides a very simple example of how the code works, but here's a step by step:
+
+1. Include all the files in the /lib directory - this includes the actual code for the parser, and a couple of different data models ('transaction' and 'account')
+
+`Dir[File.dirname(__FILE__) + '/lib/*.rb'].each {|file| require file }`
+
+2. Create a hash with three symbol keys, `:username`, `:password` and `:memorable_word`, each unsurprisingly corresponding to different authentication details used
+
+3. Instantiate a new instance of the `LloydsTSB::Customer` object, passing in the hash from the previous step - this is used to perform the authentication required.
+
+`customer = LloydsTSB::Customer.new(@settings)`
+
+4. Call the `accounts` method of the object you just made - it'll take a few seconds, and will return a number of `LloydsTSB::Account` objects. Play with the response as you wish.
+
+`customer.accounts`
+
+### Data models
+
+A __LloydsTSB::Customer__ is created with `LloydsTSB::Customer.new` with a hash of settings passed in. It has the following attributes:
+
+* __agent (Mechanize::Agent)__ - the Mechanize agent used to browse around the online banking system. This will be pointing at the "Your accounts" page.
+* __name (string)__ - the name of the customer
+* __accounts (array)__ - an array of LloydsTSB::Account objects representing accounts held by the customer
+
+A __LloydsTSB::Account__ instance has the following attributes:
+
+* __name (string)__ - the name of the account
+* __balance (integer)__ - the balance of the account, whether positive or negative. *(NB: The true meaning of balance is affected by whether the account is a :credit_card or a :bank_account)
+* __limit (integer)__ - the credit limit for the account - this is an overdraft limit for a current account, or the spending limit on a credit card
+* __transactions (array)__ - an array containing a number of `LloydsTSB::Transaction` object - this will be the 20(?) most recent transactions on the account
+* __details__ (hash)__ - the identifying information for the account as a hash. For a bank account, this will have keys :account_number and :sort_code, with :card_number for credit cards
+* __type (symbol)__ - the type of the account, either `:credit_card` or `:bank_account`
+
+A __LloydsTSB::Account__ has many __LloydsTSB::Transaction__ instances in its transactions property. Each transaction has the following attributes:
+
+* __date (Date)__ - the date of the transaction as shown on the statement
+* __narrative (string)__ - a description of the transaction, most likely the name of the merchant
+* __type (symbol)__ - the type of transaction, usually an acronym - a list is available on the Lloyds TSB site
+* __direction (symbol)__ - either `:credit` or `:debit`, depending on what the transaction is
+* __amount (integer)__ - The amount of the transaction, obviously...
+* __unique_reference (string)___ - a hash to identify this transaction *(fairly)* uniquely...useful if you want to see whether a transaction is new or not
+
+### Limitations
+
+* I haven't tested this with savings account, so it may well mess the script up and cause exceptions. I'll need to open a savings account to test this.
+* It will only show a limited number of transactions - it doesn't navigate through the different pages
+
+### License
+
+Use this for what you will, as long as it isn't evil. If you make any changes or cool improvements, please let me know at <tim+lloydstsb@tim-rogers.co.uk>.

data/example.rb

@@ -0,0 +1,30 @@
+# encoding: utf-8
+# Bring in all the files in lib/, including the scraper and the data models
+Dir[File.dirname(__FILE__) + '/lib/*.rb'].each {|file| require file }
+
+# Bring in the settings file - it should contain a hash of @settings with the
+# symbol keys :username, :password and :memorable_word
+require File.join(File.dirname(__FILE__), 'settings')
+
+# Create an instance of a Lloyds TSB customer - this is where we login.
+customer = LloydsTSB::Customer.new(@settings)
+puts "These accounts belong to #{customer.name}."
+
+customer.accounts.each do |account|
+  puts "Name: #{account.name}"
+  puts "Details: #{account.details.inspect}"
+  puts "Type: #{account.type.to_s}"
+  puts "Balance: #{currencify(account.balance)}"
+  puts "Limit: #{currencify(account.limit)}"
+  puts "Transactions:"
+  puts ""
+  account.transactions.each do |tx|
+    puts "Date: #{tx.date}"
+    puts "Description: #{tx.narrative}"
+    puts "Type: #{tx.type}"
+    puts "Direction: #{tx.direction}"
+    puts "Amount: #{currencify(tx.amount)}"
+    puts "Unique reference: #{tx.unique_reference}"
+    puts ""
+  end
+end

data/lib/lloydstsb.rb

@@ -0,0 +1,5 @@
+require 'lloydstsb/version'
+require 'lloydstsb/account'
+require 'lloydstsb/customer'
+require 'lloydstsb/transaction'
+require 'lloydstsb/utils'

data/lib/lloydstsb/account.rb

@@ -0,0 +1,10 @@
+module LloydsTSB
+  class Account
+    attr_accessor :name, :balance, :limit, :transactions, :details, :type
+    
+    def initialize(hash = {})
+      hash.each { |key,val| send("#{key}=", val) if respond_to?("#{key}=") }
+    end
+    
+  end
+end

data/lib/lloydstsb/customer.rb

@@ -0,0 +1,161 @@
+# encoding: utf-8
+require 'mechanize'
+require 'nokogiri'
+require 'open-uri'
+require 'date'
+
+module LloydsTSB
+  class Customer
+
+    attr_reader :agent, :name
+
+    def initialize(settings = {})
+      # Creates a new Customer object - expects a hash with keys :username,
+      # :password and :memorable_word
+      @agent = Mechanize.new
+      @settings = settings
+
+      if @settings[:username].blank? ||
+        @settings[:password].blank? ||
+        @settings[:memorable_word].blank?
+          raise "You must provide a username, password and memorable word."
+      end
+
+      @agent.get "https://online.lloydstsb.co.uk/personal/logon/login.jsp?WT.ac=hpIBlogon"
+      
+      # Fill in the first authentication form then submits
+      @agent.page.forms[0]["frmLogin:strCustomerLogin_userID"] = @settings[:username]
+      @agent.page.forms[0]["frmLogin:strCustomerLogin_pwd"] = @settings[:password]
+      @agent.page.forms[0].submit
+      
+      # Checks for any errors on the page indicating a failure to login
+      if @agent.page.search('.formSubmitError').any?
+        raise "There was a problem when submitting your username and password.
+          (#{@agent.page.search('.formSubmitError').text})"
+      end
+      
+      # Works out from the text on the page what characters from the memorable
+      # word are required
+      mc1 = @agent.page
+        .at('//*[@id="frmentermemorableinformation1"]/fieldset/div/div/div[1]/label').text.split(" ")[1].to_i
+      mc2 = @agent.page
+        .at('//*[@id="frmentermemorableinformation1"]/fieldset/div/div/div[2]/label').text.split(" ")[1].to_i
+      mc3 = @agent.page.
+        at('//*[@id="frmentermemorableinformation1"]/fieldset/div/div/div[3]/label')
+          .text.split(" ")[1].to_i
+        
+      # Files in the memorable word fields and logs in
+      @agent.page.forms[0]["frmentermemorableinformation1:strEnterMemorableInformation_memInfo1"] = " " + @settings[:memorable_word][mc1-1]
+      @agent.page.forms[0]["frmentermemorableinformation1:strEnterMemorableInformation_memInfo2"] = " " + @settings[:memorable_word][mc2-1]
+      @agent.page.forms[0]["frmentermemorableinformation1:strEnterMemorableInformation_memInfo3"] = " " + @settings[:memorable_word][mc3-1]
+      @agent.page.forms[0].click_button
+
+      # Checks for any errors indicating a failure to login - the final hurdle
+      if @agent.page.search('.formSubmitError').any?
+        raise "There was a problem when submitting your memorable word.
+          (#{@agent.page.search('.formSubmitError').text})"
+      end
+      
+      @name = @agent.page.at('span.name').text
+      
+    end
+
+    def accounts
+      # Fills in the relevant forms to login, gets account details and then
+      # provides a response of accounts and transactions
+      
+      return @accounts if @accounts
+
+      # We're in, now to find the accounts...
+      accounts = []
+      doc = Nokogiri::HTML(@agent.page.body, 'UTF-8')
+       doc.css('li.clearfix').each do |account|
+        # This is an account in the table - let's read out the details...
+        acct = {
+          name: account.css('a')[0].text,
+          balance: account.css('p.balance').text.split(" ")[1]
+            .gsub("£", "").gsub(",", "").to_f,
+          limit: account.css('p.accountMsg').text.split(" ")[2]
+            .gsub("£", "").gsub(",", "").to_f,
+          transactions: []
+          }
+
+        # Now we need to find the recent transactions for the account...We'll
+        # go to the account's transactions page and read the table
+        account_agent = @agent.dup
+        account_agent.get(account.css('a')[0]['href'])
+        
+        # If there's a mention of "minimum payment" on the transactions page,
+        # this is a credit card rather than a bank account
+        if account_agent.page.body.include?("Minimum payment")
+          acct[:type] = :credit_card
+          acct[:details] = {
+            card_number: account.css('.numbers').text.gsub(" Card Number ", "")
+          }
+          Nokogiri::HTML(account_agent.page.body, 'UTF-8').css('tbody tr').each do |transaction|
+            
+            # Credit card statements start with the previous statement's
+            # balance. We don't want to record this as a transaction.
+            next if transaction.css('td')[1].text == "Balance from Previous Statement"
+            
+            # Let's get the data for the transaction...
+            data = {
+              date: Date.parse(transaction.css('td')[0].text),
+              narrative: transaction.css('td')[1].text,
+            }
+            data[:amount] = transaction.css('td')[4].text.split(" ")[0].to_f
+            
+            # And now we work out whether the transaction was a credit or
+            # debit by checking, in a round-about way, whether the
+            # transaction amount contained "CR" (credit)
+            if transaction.css('td')[4].text.split(" ").length > 1
+              data[:type] = :credit
+              data[:direction] = :credit
+            else
+              data[:type] = :debit
+              data[:direction] = :debit
+            end
+            
+            # And finally, we add the transaction object to the array
+            acct[:transactions] << LloydsTSB::Transaction.new(data)
+          end
+        else
+          # This is a bank account of some description
+          acct[:type] = :bank_account
+          details = account.css('.numbers').text.gsub(" Sort Code", "").gsub("Account Number ", "").split(", ")
+          acct[:details] = {
+            sort_code: details[0],
+            account_number: details[1]
+          }
+          Nokogiri::HTML(account_agent.page.body, 'UTF-8').css('tbody tr').each do |transaction|
+            # Let's read the details from the table...
+            data = {
+              date: Date.parse(transaction.css('th.first').text),
+              narrative: transaction.css('td')[0].text,
+              type: transaction.css('td')[1].text.to_sym,
+            }
+            
+            # Regardless of what the transaction is, there's an incoming
+            # and an outgoing column. Let's work out which this is...
+            incoming = transaction.css('td')[2].text
+            out = transaction.css('td')[3].text
+            if incoming == ""
+              data[:direction] = :debit
+              data[:amount] = out.to_f
+            else
+              data[:direction] = :credit
+              data[:amount] = incoming.to_f
+            end
+            
+            # To finish, we add the newly built transaction to the array
+            acct[:transactions] << LloydsTSB::Transaction.new(data)
+          end
+        end
+
+        accounts << LloydsTSB::Account.new(acct)
+      end
+      @accounts = accounts
+      accounts
+    end
+  end
+end

data/lib/lloydstsb/transaction.rb

@@ -0,0 +1,12 @@
+require 'digest'
+
+module LloydsTSB
+  class Transaction
+    attr_accessor :date, :narrative, :type, :direction, :amount, :unique_reference
+    
+    def initialize(hash = {})
+      hash.each { |key,val| send("#{key}=", val) if respond_to?("#{key}=") }
+      @unique_reference = Digest::MD5.hexdigest("#{@date.to_s}:#{@narrative}:#{@amount}")
+    end
+  end
+end

data/lib/lloydstsb/utils.rb

@@ -0,0 +1,29 @@
+# encoding: utf-8
+class Object
+  def blank?
+    respond_to?(:empty?) ? empty? : !self
+  end
+end
+
+class Integer
+  def negative?
+    self != 0 && (self != (self * self) / self.abs)
+  end
+end
+
+def currencify(number, options={})
+  # :currency_before => false puts the currency symbol after the number
+  # default format: $12,345,678.90
+  options = {:currency_symbol => "£", :delimiter => ",", :decimal_symbol => ".", :currency_before => true}.merge(options)
+  
+  # split integer and fractional parts
+  int, frac = ("%.2f" % number).split('.')
+  # insert the delimiters
+  int.gsub!(/(\d)(?=(\d\d\d)+(?!\d))/, "\\1#{options[:delimiter]}")
+  
+  if options[:currency_before]
+    options[:currency_symbol] + int + options[:decimal_symbol] + frac
+  else
+    int + options[:decimal_symbol] + frac + options[:currency_symbol]
+  end
+end

data/lib/lloydstsb/version.rb

@@ -0,0 +1,3 @@
+module LloydsTSB
+  VERSION = '0.1.0'
+end

data/lloydstsb.gemspec

@@ -0,0 +1,14 @@
+require File.expand_path('../lib/lloydstsb/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.name = 'lloydstsb'
+  gem.version = LloydsTSB::VERSION.dup
+  gem.authors = ['Tim Rogers']
+  gem.email = ['tim@tim-rogers.co.uk']
+  gem.summary = 'A library for accessing data from Lloyds TSB\'s online banking'
+  gem.homepage = 'https://github.com/timrogers/lloydstsb'
+
+  gem.add_dependency 'mechanize', '~> 2.5.1'
+
+  gem.files = `git ls-files`.split("\n")
+end

metadata

@@ -0,0 +1,72 @@
+--- !ruby/object:Gem::Specification
+name: lloydstsb
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Tim Rogers
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-10-28 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: mechanize
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.5.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.5.1
+description: 
+email:
+- tim@tim-rogers.co.uk
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- README.md
+- example.rb
+- lib/lloydstsb.rb
+- lib/lloydstsb/account.rb
+- lib/lloydstsb/customer.rb
+- lib/lloydstsb/transaction.rb
+- lib/lloydstsb/utils.rb
+- lib/lloydstsb/version.rb
+- lloydstsb.gemspec
+homepage: https://github.com/timrogers/lloydstsb
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: A library for accessing data from Lloyds TSB's online banking
+test_files: []
+has_rdoc: