lloydstsb 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +1 -0
- data/README.md +58 -0
- data/example.rb +30 -0
- data/lib/lloydstsb.rb +5 -0
- data/lib/lloydstsb/account.rb +10 -0
- data/lib/lloydstsb/customer.rb +161 -0
- data/lib/lloydstsb/transaction.rb +12 -0
- data/lib/lloydstsb/utils.rb +29 -0
- data/lib/lloydstsb/version.rb +3 -0
- data/lloydstsb.gemspec +14 -0
- metadata +72 -0
data/.gitignore
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
settings.rb
|
data/README.md
ADDED
@@ -0,0 +1,58 @@
|
|
1
|
+
## Lloyds TSB screen scraper
|
2
|
+
|
3
|
+
I bank with Lloyds TSB - I have my current account and credit card with them. Like most online banking services though, they're not to up-to-date on APIs and the like. After looking around online, I found that there were a couple of scripts that people had built, but I didn't have much luck with them myself. So I decided to build my own screen scraper.
|
4
|
+
|
5
|
+
I know the code in this is pretty messy, and as ever, it's untested. I tried to refactor it and got to the end, but then it turned out to be broken and I couldn't be bothered to fix it. So I've left it for now.
|
6
|
+
|
7
|
+
### Usage
|
8
|
+
|
9
|
+
The file `example.rb` provides a very simple example of how the code works, but here's a step by step:
|
10
|
+
|
11
|
+
1. Include all the files in the /lib directory - this includes the actual code for the parser, and a couple of different data models ('transaction' and 'account')
|
12
|
+
|
13
|
+
`Dir[File.dirname(__FILE__) + '/lib/*.rb'].each {|file| require file }`
|
14
|
+
|
15
|
+
2. Create a hash with three symbol keys, `:username`, `:password` and `:memorable_word`, each unsurprisingly corresponding to different authentication details used
|
16
|
+
|
17
|
+
3. Instantiate a new instance of the `LloydsTSB::Customer` object, passing in the hash from the previous step - this is used to perform the authentication required.
|
18
|
+
|
19
|
+
`customer = LloydsTSB::Customer.new(@settings)`
|
20
|
+
|
21
|
+
4. Call the `accounts` method of the object you just made - it'll take a few seconds, and will return a number of `LloydsTSB::Account` objects. Play with the response as you wish.
|
22
|
+
|
23
|
+
`customer.accounts`
|
24
|
+
|
25
|
+
### Data models
|
26
|
+
|
27
|
+
A __LloydsTSB::Customer__ is created with `LloydsTSB::Customer.new` with a hash of settings passed in. It has the following attributes:
|
28
|
+
|
29
|
+
* __agent (Mechanize::Agent)__ - the Mechanize agent used to browse around the online banking system. This will be pointing at the "Your accounts" page.
|
30
|
+
* __name (string)__ - the name of the customer
|
31
|
+
* __accounts (array)__ - an array of LloydsTSB::Account objects representing accounts held by the customer
|
32
|
+
|
33
|
+
A __LloydsTSB::Account__ instance has the following attributes:
|
34
|
+
|
35
|
+
* __name (string)__ - the name of the account
|
36
|
+
* __balance (integer)__ - the balance of the account, whether positive or negative. *(NB: The true meaning of balance is affected by whether the account is a :credit_card or a :bank_account)
|
37
|
+
* __limit (integer)__ - the credit limit for the account - this is an overdraft limit for a current account, or the spending limit on a credit card
|
38
|
+
* __transactions (array)__ - an array containing a number of `LloydsTSB::Transaction` object - this will be the 20(?) most recent transactions on the account
|
39
|
+
* __details__ (hash)__ - the identifying information for the account as a hash. For a bank account, this will have keys :account_number and :sort_code, with :card_number for credit cards
|
40
|
+
* __type (symbol)__ - the type of the account, either `:credit_card` or `:bank_account`
|
41
|
+
|
42
|
+
A __LloydsTSB::Account__ has many __LloydsTSB::Transaction__ instances in its transactions property. Each transaction has the following attributes:
|
43
|
+
|
44
|
+
* __date (Date)__ - the date of the transaction as shown on the statement
|
45
|
+
* __narrative (string)__ - a description of the transaction, most likely the name of the merchant
|
46
|
+
* __type (symbol)__ - the type of transaction, usually an acronym - a list is available on the Lloyds TSB site
|
47
|
+
* __direction (symbol)__ - either `:credit` or `:debit`, depending on what the transaction is
|
48
|
+
* __amount (integer)__ - The amount of the transaction, obviously...
|
49
|
+
* __unique_reference (string)___ - a hash to identify this transaction *(fairly)* uniquely...useful if you want to see whether a transaction is new or not
|
50
|
+
|
51
|
+
### Limitations
|
52
|
+
|
53
|
+
* I haven't tested this with savings account, so it may well mess the script up and cause exceptions. I'll need to open a savings account to test this.
|
54
|
+
* It will only show a limited number of transactions - it doesn't navigate through the different pages
|
55
|
+
|
56
|
+
### License
|
57
|
+
|
58
|
+
Use this for what you will, as long as it isn't evil. If you make any changes or cool improvements, please let me know at <tim+lloydstsb@tim-rogers.co.uk>.
|
data/example.rb
ADDED
@@ -0,0 +1,30 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
# Bring in all the files in lib/, including the scraper and the data models
|
3
|
+
Dir[File.dirname(__FILE__) + '/lib/*.rb'].each {|file| require file }
|
4
|
+
|
5
|
+
# Bring in the settings file - it should contain a hash of @settings with the
|
6
|
+
# symbol keys :username, :password and :memorable_word
|
7
|
+
require File.join(File.dirname(__FILE__), 'settings')
|
8
|
+
|
9
|
+
# Create an instance of a Lloyds TSB customer - this is where we login.
|
10
|
+
customer = LloydsTSB::Customer.new(@settings)
|
11
|
+
puts "These accounts belong to #{customer.name}."
|
12
|
+
|
13
|
+
customer.accounts.each do |account|
|
14
|
+
puts "Name: #{account.name}"
|
15
|
+
puts "Details: #{account.details.inspect}"
|
16
|
+
puts "Type: #{account.type.to_s}"
|
17
|
+
puts "Balance: #{currencify(account.balance)}"
|
18
|
+
puts "Limit: #{currencify(account.limit)}"
|
19
|
+
puts "Transactions:"
|
20
|
+
puts ""
|
21
|
+
account.transactions.each do |tx|
|
22
|
+
puts "Date: #{tx.date}"
|
23
|
+
puts "Description: #{tx.narrative}"
|
24
|
+
puts "Type: #{tx.type}"
|
25
|
+
puts "Direction: #{tx.direction}"
|
26
|
+
puts "Amount: #{currencify(tx.amount)}"
|
27
|
+
puts "Unique reference: #{tx.unique_reference}"
|
28
|
+
puts ""
|
29
|
+
end
|
30
|
+
end
|
data/lib/lloydstsb.rb
ADDED
@@ -0,0 +1,161 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
require 'mechanize'
|
3
|
+
require 'nokogiri'
|
4
|
+
require 'open-uri'
|
5
|
+
require 'date'
|
6
|
+
|
7
|
+
module LloydsTSB
|
8
|
+
class Customer
|
9
|
+
|
10
|
+
attr_reader :agent, :name
|
11
|
+
|
12
|
+
def initialize(settings = {})
|
13
|
+
# Creates a new Customer object - expects a hash with keys :username,
|
14
|
+
# :password and :memorable_word
|
15
|
+
@agent = Mechanize.new
|
16
|
+
@settings = settings
|
17
|
+
|
18
|
+
if @settings[:username].blank? ||
|
19
|
+
@settings[:password].blank? ||
|
20
|
+
@settings[:memorable_word].blank?
|
21
|
+
raise "You must provide a username, password and memorable word."
|
22
|
+
end
|
23
|
+
|
24
|
+
@agent.get "https://online.lloydstsb.co.uk/personal/logon/login.jsp?WT.ac=hpIBlogon"
|
25
|
+
|
26
|
+
# Fill in the first authentication form then submits
|
27
|
+
@agent.page.forms[0]["frmLogin:strCustomerLogin_userID"] = @settings[:username]
|
28
|
+
@agent.page.forms[0]["frmLogin:strCustomerLogin_pwd"] = @settings[:password]
|
29
|
+
@agent.page.forms[0].submit
|
30
|
+
|
31
|
+
# Checks for any errors on the page indicating a failure to login
|
32
|
+
if @agent.page.search('.formSubmitError').any?
|
33
|
+
raise "There was a problem when submitting your username and password.
|
34
|
+
(#{@agent.page.search('.formSubmitError').text})"
|
35
|
+
end
|
36
|
+
|
37
|
+
# Works out from the text on the page what characters from the memorable
|
38
|
+
# word are required
|
39
|
+
mc1 = @agent.page
|
40
|
+
.at('//*[@id="frmentermemorableinformation1"]/fieldset/div/div/div[1]/label').text.split(" ")[1].to_i
|
41
|
+
mc2 = @agent.page
|
42
|
+
.at('//*[@id="frmentermemorableinformation1"]/fieldset/div/div/div[2]/label').text.split(" ")[1].to_i
|
43
|
+
mc3 = @agent.page.
|
44
|
+
at('//*[@id="frmentermemorableinformation1"]/fieldset/div/div/div[3]/label')
|
45
|
+
.text.split(" ")[1].to_i
|
46
|
+
|
47
|
+
# Files in the memorable word fields and logs in
|
48
|
+
@agent.page.forms[0]["frmentermemorableinformation1:strEnterMemorableInformation_memInfo1"] = " " + @settings[:memorable_word][mc1-1]
|
49
|
+
@agent.page.forms[0]["frmentermemorableinformation1:strEnterMemorableInformation_memInfo2"] = " " + @settings[:memorable_word][mc2-1]
|
50
|
+
@agent.page.forms[0]["frmentermemorableinformation1:strEnterMemorableInformation_memInfo3"] = " " + @settings[:memorable_word][mc3-1]
|
51
|
+
@agent.page.forms[0].click_button
|
52
|
+
|
53
|
+
# Checks for any errors indicating a failure to login - the final hurdle
|
54
|
+
if @agent.page.search('.formSubmitError').any?
|
55
|
+
raise "There was a problem when submitting your memorable word.
|
56
|
+
(#{@agent.page.search('.formSubmitError').text})"
|
57
|
+
end
|
58
|
+
|
59
|
+
@name = @agent.page.at('span.name').text
|
60
|
+
|
61
|
+
end
|
62
|
+
|
63
|
+
def accounts
|
64
|
+
# Fills in the relevant forms to login, gets account details and then
|
65
|
+
# provides a response of accounts and transactions
|
66
|
+
|
67
|
+
return @accounts if @accounts
|
68
|
+
|
69
|
+
# We're in, now to find the accounts...
|
70
|
+
accounts = []
|
71
|
+
doc = Nokogiri::HTML(@agent.page.body, 'UTF-8')
|
72
|
+
doc.css('li.clearfix').each do |account|
|
73
|
+
# This is an account in the table - let's read out the details...
|
74
|
+
acct = {
|
75
|
+
name: account.css('a')[0].text,
|
76
|
+
balance: account.css('p.balance').text.split(" ")[1]
|
77
|
+
.gsub("£", "").gsub(",", "").to_f,
|
78
|
+
limit: account.css('p.accountMsg').text.split(" ")[2]
|
79
|
+
.gsub("£", "").gsub(",", "").to_f,
|
80
|
+
transactions: []
|
81
|
+
}
|
82
|
+
|
83
|
+
# Now we need to find the recent transactions for the account...We'll
|
84
|
+
# go to the account's transactions page and read the table
|
85
|
+
account_agent = @agent.dup
|
86
|
+
account_agent.get(account.css('a')[0]['href'])
|
87
|
+
|
88
|
+
# If there's a mention of "minimum payment" on the transactions page,
|
89
|
+
# this is a credit card rather than a bank account
|
90
|
+
if account_agent.page.body.include?("Minimum payment")
|
91
|
+
acct[:type] = :credit_card
|
92
|
+
acct[:details] = {
|
93
|
+
card_number: account.css('.numbers').text.gsub(" Card Number ", "")
|
94
|
+
}
|
95
|
+
Nokogiri::HTML(account_agent.page.body, 'UTF-8').css('tbody tr').each do |transaction|
|
96
|
+
|
97
|
+
# Credit card statements start with the previous statement's
|
98
|
+
# balance. We don't want to record this as a transaction.
|
99
|
+
next if transaction.css('td')[1].text == "Balance from Previous Statement"
|
100
|
+
|
101
|
+
# Let's get the data for the transaction...
|
102
|
+
data = {
|
103
|
+
date: Date.parse(transaction.css('td')[0].text),
|
104
|
+
narrative: transaction.css('td')[1].text,
|
105
|
+
}
|
106
|
+
data[:amount] = transaction.css('td')[4].text.split(" ")[0].to_f
|
107
|
+
|
108
|
+
# And now we work out whether the transaction was a credit or
|
109
|
+
# debit by checking, in a round-about way, whether the
|
110
|
+
# transaction amount contained "CR" (credit)
|
111
|
+
if transaction.css('td')[4].text.split(" ").length > 1
|
112
|
+
data[:type] = :credit
|
113
|
+
data[:direction] = :credit
|
114
|
+
else
|
115
|
+
data[:type] = :debit
|
116
|
+
data[:direction] = :debit
|
117
|
+
end
|
118
|
+
|
119
|
+
# And finally, we add the transaction object to the array
|
120
|
+
acct[:transactions] << LloydsTSB::Transaction.new(data)
|
121
|
+
end
|
122
|
+
else
|
123
|
+
# This is a bank account of some description
|
124
|
+
acct[:type] = :bank_account
|
125
|
+
details = account.css('.numbers').text.gsub(" Sort Code", "").gsub("Account Number ", "").split(", ")
|
126
|
+
acct[:details] = {
|
127
|
+
sort_code: details[0],
|
128
|
+
account_number: details[1]
|
129
|
+
}
|
130
|
+
Nokogiri::HTML(account_agent.page.body, 'UTF-8').css('tbody tr').each do |transaction|
|
131
|
+
# Let's read the details from the table...
|
132
|
+
data = {
|
133
|
+
date: Date.parse(transaction.css('th.first').text),
|
134
|
+
narrative: transaction.css('td')[0].text,
|
135
|
+
type: transaction.css('td')[1].text.to_sym,
|
136
|
+
}
|
137
|
+
|
138
|
+
# Regardless of what the transaction is, there's an incoming
|
139
|
+
# and an outgoing column. Let's work out which this is...
|
140
|
+
incoming = transaction.css('td')[2].text
|
141
|
+
out = transaction.css('td')[3].text
|
142
|
+
if incoming == ""
|
143
|
+
data[:direction] = :debit
|
144
|
+
data[:amount] = out.to_f
|
145
|
+
else
|
146
|
+
data[:direction] = :credit
|
147
|
+
data[:amount] = incoming.to_f
|
148
|
+
end
|
149
|
+
|
150
|
+
# To finish, we add the newly built transaction to the array
|
151
|
+
acct[:transactions] << LloydsTSB::Transaction.new(data)
|
152
|
+
end
|
153
|
+
end
|
154
|
+
|
155
|
+
accounts << LloydsTSB::Account.new(acct)
|
156
|
+
end
|
157
|
+
@accounts = accounts
|
158
|
+
accounts
|
159
|
+
end
|
160
|
+
end
|
161
|
+
end
|
@@ -0,0 +1,12 @@
|
|
1
|
+
require 'digest'
|
2
|
+
|
3
|
+
module LloydsTSB
|
4
|
+
class Transaction
|
5
|
+
attr_accessor :date, :narrative, :type, :direction, :amount, :unique_reference
|
6
|
+
|
7
|
+
def initialize(hash = {})
|
8
|
+
hash.each { |key,val| send("#{key}=", val) if respond_to?("#{key}=") }
|
9
|
+
@unique_reference = Digest::MD5.hexdigest("#{@date.to_s}:#{@narrative}:#{@amount}")
|
10
|
+
end
|
11
|
+
end
|
12
|
+
end
|
@@ -0,0 +1,29 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
class Object
|
3
|
+
def blank?
|
4
|
+
respond_to?(:empty?) ? empty? : !self
|
5
|
+
end
|
6
|
+
end
|
7
|
+
|
8
|
+
class Integer
|
9
|
+
def negative?
|
10
|
+
self != 0 && (self != (self * self) / self.abs)
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
def currencify(number, options={})
|
15
|
+
# :currency_before => false puts the currency symbol after the number
|
16
|
+
# default format: $12,345,678.90
|
17
|
+
options = {:currency_symbol => "£", :delimiter => ",", :decimal_symbol => ".", :currency_before => true}.merge(options)
|
18
|
+
|
19
|
+
# split integer and fractional parts
|
20
|
+
int, frac = ("%.2f" % number).split('.')
|
21
|
+
# insert the delimiters
|
22
|
+
int.gsub!(/(\d)(?=(\d\d\d)+(?!\d))/, "\\1#{options[:delimiter]}")
|
23
|
+
|
24
|
+
if options[:currency_before]
|
25
|
+
options[:currency_symbol] + int + options[:decimal_symbol] + frac
|
26
|
+
else
|
27
|
+
int + options[:decimal_symbol] + frac + options[:currency_symbol]
|
28
|
+
end
|
29
|
+
end
|
data/lloydstsb.gemspec
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
require File.expand_path('../lib/lloydstsb/version', __FILE__)
|
2
|
+
|
3
|
+
Gem::Specification.new do |gem|
|
4
|
+
gem.name = 'lloydstsb'
|
5
|
+
gem.version = LloydsTSB::VERSION.dup
|
6
|
+
gem.authors = ['Tim Rogers']
|
7
|
+
gem.email = ['tim@tim-rogers.co.uk']
|
8
|
+
gem.summary = 'A library for accessing data from Lloyds TSB\'s online banking'
|
9
|
+
gem.homepage = 'https://github.com/timrogers/lloydstsb'
|
10
|
+
|
11
|
+
gem.add_dependency 'mechanize', '~> 2.5.1'
|
12
|
+
|
13
|
+
gem.files = `git ls-files`.split("\n")
|
14
|
+
end
|
metadata
ADDED
@@ -0,0 +1,72 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: lloydstsb
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Tim Rogers
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2012-10-28 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: mechanize
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ~>
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: 2.5.1
|
22
|
+
type: :runtime
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ~>
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 2.5.1
|
30
|
+
description:
|
31
|
+
email:
|
32
|
+
- tim@tim-rogers.co.uk
|
33
|
+
executables: []
|
34
|
+
extensions: []
|
35
|
+
extra_rdoc_files: []
|
36
|
+
files:
|
37
|
+
- .gitignore
|
38
|
+
- README.md
|
39
|
+
- example.rb
|
40
|
+
- lib/lloydstsb.rb
|
41
|
+
- lib/lloydstsb/account.rb
|
42
|
+
- lib/lloydstsb/customer.rb
|
43
|
+
- lib/lloydstsb/transaction.rb
|
44
|
+
- lib/lloydstsb/utils.rb
|
45
|
+
- lib/lloydstsb/version.rb
|
46
|
+
- lloydstsb.gemspec
|
47
|
+
homepage: https://github.com/timrogers/lloydstsb
|
48
|
+
licenses: []
|
49
|
+
post_install_message:
|
50
|
+
rdoc_options: []
|
51
|
+
require_paths:
|
52
|
+
- lib
|
53
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
54
|
+
none: false
|
55
|
+
requirements:
|
56
|
+
- - ! '>='
|
57
|
+
- !ruby/object:Gem::Version
|
58
|
+
version: '0'
|
59
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
60
|
+
none: false
|
61
|
+
requirements:
|
62
|
+
- - ! '>='
|
63
|
+
- !ruby/object:Gem::Version
|
64
|
+
version: '0'
|
65
|
+
requirements: []
|
66
|
+
rubyforge_project:
|
67
|
+
rubygems_version: 1.8.24
|
68
|
+
signing_key:
|
69
|
+
specification_version: 3
|
70
|
+
summary: A library for accessing data from Lloyds TSB's online banking
|
71
|
+
test_files: []
|
72
|
+
has_rdoc:
|