bankjob 0.5.0

data/History.txt

@@ -0,0 +1,4 @@
+== 0.0.1 2009-02-13
+
+* 1 major enhancement:
+  * Initial release

data/PostInstall.txt

@@ -0,0 +1,4 @@
+
+For more information on bankjob, see http://bankjob.rubyforge.org
+
+

data/README.rdoc

@@ -0,0 +1,77 @@
+= bankjob
+
+http://bankjob.rubyforge.org/
+
+== DESCRIPTION:
+
+Bankjob is a command-line ruby program for scraping online banking sites and producing statements in OFX (Open Fincancial Exchange) or CSV (Comma Separated Values) formats.
+
+Bankjob was created for people like me who want to get their bank data into a 3rd party application but whose bank does not support downloads in OFX format.
+It's also useful for keeping a permanent store of bank statements on your computer for reading in Excel (vs filing paper statements)
+
+== FEATURES:
+
+* Scrapes an online banking website to produce a bank statement
+* Stores bank statements locally in CSV files, which can be loaded directly in spreadsheets like Microsoft Excel
+* Stores bank statements locally in OFX files, which can be imported by many programs such as Quicken, MS Money, Gnu Cash and uploaded to some web applications
+* Built-in support for uploading to your Wesabe account (www.wesabe.com)
+* Supports coding of simple rules in ruby for modifying transaction details. E.g. automatically change "pment inst 3245003" to "paid home loan interest"
+
+== SYNOPSIS:
+
+  bankjob --csv c:\bank\csv --scraper c:\bank\my_bpi_scraper.rb
+          --scraper-args "<my_username> <my_password>"
+          --wesabe "<wesabe_user> <wesabe_pass> <wesabe_acct>
+          --log c:\bank\bankjob.log --debug
+
+I have this command in a .bat file which is launched daily by a scheduled task on my windows Media Center PC (which, since it's always on and connected to the internet, makes a useful home server)
+
+This one command will:
+* scrape my online banking website after logging in as me and navigating to the page with recent transactions
+* apply some rules, coded in the my_bpi_scraper.rb file that make the descriptions more readable
+* produce a statement in comma-separated-value format, keeping the original raw data as well as the new descriptions,
+  storing that in a file with a name like "20090327-20090406.csv" in my local directory c:\bank\csv (a permanent record)
+* produce an OFX document with the same statement information
+* upload the OFX statement to my wesabe account
+* log debug-level details in bankjob.log
+
+== REQUIREMENTS:
+
+* Runs in ruby so you need to have ruby installed
+* Requires a scraper for your online bank site
+  Some examples come packaged with Bankjob but you will probably need to write your own scraper in ruby.
+  For help go to http://groups.google.com/group/bankjob, but read http://bankjob.rubyforge.org first.
+
+== INSTALL:
+
+Mac OSX (linux):
+
+<tt>sudo gem install bankjob</tt>
+
+Windows:
+<tt>gem install bankjob</tt>
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2009 rubarb.bankjob@gmail.com
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/bin/bankjob

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+#
+#  Created on 2009-2-13.
+#  Copyright (c) 2009. All rights reserved.
+
+require File.expand_path(File.dirname(__FILE__) + "/../lib/bankjob")
+
+require "bankjob/cli"
+
+Bankjob::CLI.execute(STDOUT, ARGV)

data/lib/bankjob.rb

@@ -0,0 +1,12 @@
+$:.unshift(File.dirname(__FILE__)) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+
+require 'bankjob/support.rb'
+require 'bankjob/statement.rb'
+require 'bankjob/transaction.rb'
+require 'bankjob/scraper.rb'
+require 'bankjob/payee.rb'
+
+module Bankjob
+  BANKJOB_VERSION = '0.5.0' unless defined?(BANKJOB_VERSION)
+end

data/lib/bankjob/bankjob_runner.rb

@@ -0,0 +1,184 @@
+require 'rubygems'
+require 'logger'
+require 'bankjob.rb'
+
+module Bankjob
+  class BankjobRunner
+
+    # Runs the bankjob application, loading and running the
+    # scraper specified in the command line args and generating
+    # the output file.
+    def run(options, stdout)
+      logger = options.logger
+
+      if options.wesabe_help
+        Bankjob.wesabe_help(options.wesabe_args)
+        exit(0) # Wesabe help describes to the user how to use the wesabe options then quits
+      end
+
+      # Load the scraper object dynamically, then scrape the web
+      # to get a new bank statement
+      scraper = Scraper.load_scraper(options.scraper, options, logger)
+
+      begin
+        statement = scraper.scrape_statement(options.scraper_args)
+        statement = Scraper.post_process_transactions(statement)
+      rescue Exception => e
+        logger.fatal(e)
+        puts "Failed to scrape a statement successfully with #{options.scraper} due to: #{e.message}\n"
+        puts "Use --debug --log bankjob.log then check the log for more details"
+        exit (1)
+      end
+
+      # a lot of if's here but we allow for the user to generate ofx
+      # and csv to files while simultaneously uploading to wesabe
+
+      if options.csv
+        if options.csv_out.nil?
+          puts write_csv_doc([statement], true) # dump to console with header, no file specified
+        else
+          csv_file = file_name_from_option(options.csv_out, statement, "csv")
+
+          # Output data as comma separated values possibly merging
+          if File.file?(csv_file)
+            # TODO until we fix merging csv files are appended
+            open(csv_file, "a") do |f|
+              f.puts(write_csv_doc([statement]))
+            end
+            logger.info("Statement is being appended as csv to #{csv_file}")
+            #
+            # TODO fix the merging then uncomment this
+#            old_file_path = csv_file
+#            # The file already exists, lets load it and merge with the new data
+#            old_statement = scraper.create_statement()
+#            old_statement.from_csv(old_file_path, scraper.decimal)
+#            begin
+#              old_statement.merge!(statement)
+#              statement = old_statement
+#            rescue Exception => e
+#              # the merge failed, so leave the statement as the original and store it separately
+#              output_file = output_file + "_#{date_range}_merge_failed"
+#              logger.warn("Merge failed, storing new data in #{output_file} instead of appending it to #{old_file_path}")
+#              logger.debug("Merge failed due to: #{e.message}")
+#            end
+          else
+            open(csv_file, "w") do |f|
+              f.puts(write_csv_doc([statement], true)) # true = write with header
+            end
+            logger.info("Statement is being written as csv to #{csv_file}")
+          end
+        end
+      end # if csv
+
+      # Create an ofx document and write it if necessary
+      if (options.ofx or options.wesabe_upload)
+        ofx_doc = write_ofx_doc([statement])
+      end
+
+      # Output ofx file
+      if options.ofx
+        if options.ofx_out.nil?
+          puts ofx_doc # dump to console, no file specified
+        else
+          ofx_file = file_name_from_option(options.ofx_out, statement, "ofx")
+          open(ofx_file, "w") do |f|
+            f.puts(ofx_doc)
+          end
+          logger.info("Statement is being output as ofx to #{ofx_file}")
+        end
+      end
+
+      # Upload to wesabe if requested
+      if options.wesabe_upload
+        begin
+          Bankjob.wesabe_upload(options.wesabe_args, ofx_doc, logger)
+        rescue Exception => e
+          logger.fatal("Failed to upload to Wesabe")
+          logger.fatal(e)
+          puts "Failed to upload to Wesabe: #{e.message}\n"
+          puts "Try bankjob --wesabe-help for help on this feature."
+          exit(1)
+        end
+      end
+    end # run
+
+    ##
+    # Generates an OFX document to a string that starts with the stanadard
+    # OFX header and contains the XML for the specified +statements+
+    #
+    def write_ofx_doc(statements)
+      ofx = generate_ofx2_header
+      statements.each do |statement|
+        ofx << statement.to_ofx
+      end
+      return ofx
+    end
+ 
+    ##
+    # Generates a CSV document to a string containing the transactions in
+    # all of the specified +statements+
+    #
+    def write_csv_doc(statements, header = false)
+      csv = ""
+      csv << Statement.csv_header if header
+      statements.each do |statement|
+        csv << statement.to_csv
+      end
+      return csv
+    end
+
+    ##
+    # Generates the (XML) OFX2 header lines that allow the OFX 2.0 document
+    # to be recognized.
+    #
+    # <em>(Note that this is crucial for www.wesabe.com to accept the OFX
+    # document in an upload)</em>
+    #
+    def generate_ofx2_header
+      return <<-EOF
+<?xml version="1.0" encoding="UTF-8"?>
+<?OFX OFXHEADER="200" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="NONE" VERSION="200"?>
+      EOF
+    end
+
+    ##
+    # Generates the (non-XML) OFX header lines that allow the OFX 1.0 document
+    # to be recognized.
+    #
+    # <em>(Note that this is crucial for www.wesabe.com to accept the OFX
+    # document in an upload)</em>
+    #
+    def generate_ofx_header
+      return <<-EOF
+OFXHEADER:100
+DATA:OFXSGML
+VERSION:102
+SECURITY:NONE
+ENCODING:USASCII
+CHARSET:1252
+COMPRESSION:NONE
+OLDFILEUID:NONE
+NEWFILEUID:NONE
+      EOF
+    end
+
+    ##
+    # Takes a name or path for an output file and a Statement and if the file
+    # path is a directory, creates a new file name based on the date range
+    # of the statement and returns a path to that file.
+    # If +output_file+ is not a directory it is returned as-is.
+    #
+    def file_name_from_option(output_file, statement, type)
+      # if the output_file is a directory, we create a new file name 
+      if (output_file and File.directory?(output_file))
+        # Create a date range string for the first and last transactions in the statement
+        # This will looks something like: 20090130000000-20090214000000
+        date_range = "#{Bankjob.date_time_to_ofx(statement.from_date)[0..7]}-#{Bankjob.date_time_to_ofx(statement.to_date)[0..7]}"
+        filename = "#{date_range}.#{type}"
+        output_file = File.join(output_file, filename)
+      end
+      # else we assume output_file is a file name/path already
+      return output_file
+    end
+  end # class BankjobRunner
+end # module Bankjob

data/lib/bankjob/cli.rb

@@ -0,0 +1,258 @@
+require 'rubygems'
+require 'ostruct'
+require 'optparse'
+require 'logger'
+
+$:.unshift(File.dirname(__FILE__)) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+
+require 'bankjob_runner.rb'
+
+module Bankjob
+  class CLI
+
+    NEEDED = "Needed" # constant to indicate compulsory options
+    NOT_NEEDED = "Not Needed" # constant to indicate no-longer compulsory options
+
+    def self.execute(stdout, argv)
+      # The BanjobOptions module above, through the magic of OptiFlags
+      # has augmented ARGV with the command line options accessible through
+      # ARGV.flags.
+      runner = BankjobRunner.new()
+      runner.run(parse(argv), stdout)
+    end # execute
+
+    ##
+    # Parses the command line arguments using OptionParser and returns
+    # an open struct with an attribute for each option
+    #
+    def self.parse(args)
+      options = OpenStruct.new
+
+      # Set the default options
+      options.scraper = NEEDED
+      options.scraper_args = []
+      options.log_level = Logger::WARN
+      options.log_file = nil
+      options.debug = false
+      options.input = nil
+      options.ofx = false # ofx is the default but only if csv is false
+      options.ofx_out = false
+      options.csv = false
+      options.csv_out = nil # allow for separate csv and ofx output files
+      options.wesabe_help = false
+      options.wesabe_upload = false
+      options.wesabe_args = nil
+      options.logger = nil
+
+      opt = OptionParser.new do |opt|
+  
+        opt.banner = "Bankjob - scrapes your online banking website and produces an OFX or CSV document.\n" +
+                     "Usage: bankjob [options]\n"
+
+        opt.version = Bankjob::BANKJOB_VERSION
+  
+        opt.on('-s', '--scraper SCRAPER',
+               "The name of the ruby file that scrapes the website.\n") do |file|
+          options.scraper = file
+        end
+
+        opt.on('--scraper-args ARGS',
+               "Any arguments you want to pass on to your scraper.",
+               "The entire set of arguments must be quoted and separated by spaces",
+               "but you can use single quotes to specify multi-word arguments for",
+               "your scraper.  E.g.",
+               "   -scraper-args \"-user Joe -password Joe123 -arg3 'two words'\""," ",
+               "This assumes your scraper accepts an array of args and knows what",
+               "to do with them, it will vary from scraper to scraper.\n") do |sargs|
+          options.scraper_args = sub_args_to_array(sargs)
+        end
+
+        opt.on('-i', '--input INPUT_HTML_FILE',
+               "An html file used as the input instead of scraping the website -",
+               "useful for debugging.\n") do |file|
+          options.input = file
+        end
+
+        opt.on('-l', '--log LOG_FILE',
+               "Specify a file to log information and debug messages.",
+               "If --debug is used, log info will go to the console, but if neither",
+               "this nor --debug is specfied, there will be no log.",
+               "Note that the log is rolled over once per week\n") do |log_file|
+          options.log_file = log_file
+        end
+
+        opt.on('q', '--quiet', "Suppress all messages, warnings and errors.",
+               "Only fatal errors will go in the log") do
+          options.log_level = Logger::FATAL
+        end
+
+        opt.on( '--verbose', "Log detailed informational messages.\n") do
+          options.log_level = Logger::INFO
+        end
+
+        opt.on('--debug',
+               "Log debug-level information to the log",
+               "if here is one and put debug info in log\n") do
+          options.log_level = Logger::DEBUG
+          options.debug = true
+        end
+
+        opt.on('--ofx [FILE]',
+               "Write out the statement as an OFX2 compliant XML document."," ",
+               "If FILE is not specified, the XML is dumped to the console.",
+               "If FILE specifies a directory then a new file will be created with a",
+               "name generated from the dates of the first and last transactions.",
+               "If FILE specifies a file that already exists it will be overwritten."," ",
+               "(Note that ofx is the default format unless --csv is specified,",
+               "and that both CSV and OFX documents can be produced by specifying",
+               "both options.)\n") do |file|
+          options.ofx = true
+          options.ofx_out = file
+        end
+
+        opt.on('--csv [FILE]',
+               "Writes out the statement as a CSV (comma separated values) document.",
+               "All of the information available including numeric values for amount,",
+               "raw and rule-generated descriptions, etc, are produced in the CSV document.", " ",
+               "The document produced is suitable for loading into a spreadsheet like",
+               "Microsoft Excel with the dates formatted to allow for auto recognition.",
+               "This option can be used in conjunction with --ofx or --wesabe to produce",
+               "a local permanent log of all the data scraped over time.", " ",
+               "If FILE is not specified, the CSV is dumped to the console.",
+               "If FILE specifies a directory then a new file will be created with a",
+               "name generated from the dates of the first and last transactions.",
+               "If FILE specifies a file that already exists then the new statement",
+               "will be appended to the existing one in that file with care taken to",
+               "merge removing duplicate entries.\n",
+               "[WARNING - this merging does not yet function properly - its best to specify a directory for now.]\n"
+             ) do |file|
+          # TODO update this warning when we have merging working
+          options.csv = true
+          options.csv_out = file
+        end
+
+        opt.on('--wesabe-help [WESABE_ARGS]',
+               "Show help information on how to use Bankjob to upload to Wesabe.",
+               "Optionally use with \"wesabe-user password\" to get Wesabe account info.",
+               "Note that the quotes around the WESABE_ARGS to send both username",
+               "and password are necessary.", " ",
+               "Use --wesabe-help with no args for more details.\n") do |wargs|
+          options.wesabe_args = sub_args_to_array(wargs)
+          options.wesabe_help = true
+          options.scraper = NOT_NEEDED # scraper is not NEEDED when this option is set
+        end
+
+        opt.on('--wesabe WESABE_ARGS',
+               "Produce an OFX document from the statement and upload it to a Wesabe account.",
+               "WESABE_ARGS must be quoted and space-separated, specifying the wesabe account",
+               "username, password and - if there is more than one - the wesabe account number.", " ",
+               "Before trying this, use bankjob --wesabe-help to get more information.\n"
+             ) do |wargs|
+          options.wesabe_args = sub_args_to_array(wargs)
+          options.wesabe_upload = true
+        end
+
+        opt.on('--version', "Display program version and exit.\n" ) do
+          puts opt.version
+          exit
+        end
+ 
+        opt.on_tail('-h', '--help', "Display this usage message and exit.\n" ) do
+          puts opt
+          puts <<-EOF
+
+  Some common options:
+
+    o Debugging:
+      --debug --scraper bpi_scraper.rb --input /tmp/DownloadedPage.html --ofx
+     
+    o Regular use: (output ofx documents to a directory called 'bank')
+      --scraper /bank/mybank_scraper.rb --scraper-args "me mypass123" --ofx /bank --log /bank/bankjob.log --verbose
+      
+    o Abbreviated options with CSV output: (output csv appended continuously to a file)
+      -s /bank/otherbank_scraper.rb --csv /bank/statements.csv -l /bank/bankjob.log -q
+
+    o Get help on using Wesabe:
+      --wesabe-help
+
+    o Upload to Wesabe: (I have 4 Wesabe accounts and am uploading to the 3rd)
+      -s /bank/mybank_scraper.rb --wesabe "mywesabeuser password 3"  -l /bank/bankjob.log --debug
+  EOF
+          exit!
+        end
+  
+      end #OptionParser.new
+
+      begin
+        opt.parse!(args)
+        _validate_options(options) # will raise exceptions if options are invalid
+        _init_logger(options) # sets the logger
+      rescue Exception => e
+        puts e, "", opt
+        exit
+      end
+  
+      return options
+    end #self.parse
+
+    private
+
+    # Checks if the options are valid, raising exceptiosn if they are not.
+    # If the --debug option is true, then messages are dumped but flow continues
+    def self._validate_options(options)
+      begin 
+        #Note that OptionParser doesn't really handle compulsory arguments so we use
+        #our own mechanism
+        if options.scraper == NEEDED
+          raise "Incomplete arguments: You must specify a scaper ruby script with --scraper"
+        end
+
+        # Add in the --ofx option if it is not already specified and if --csv is not specified either
+        options.ofx = true unless options.csv or options.wesabe_upload
+      rescue Exception => e
+        if options.debug
+          # just dump the message and eat the exception - 
+          # we may be using dummy values for debugging
+          puts "Ignoring error in options due to --debug flag: #{e}"
+        else
+          raise e
+        end
+      end #begin/rescue
+
+    end #_validate_options
+
+    ##
+    # Initializes the logger taking the log-level and the log
+    # file name from the command line +options+ and setting the logger back on
+    # the options struct as +options.logger+
+    #
+    # Note that the level is not set explicitly in options but derived from
+    # flag options like --verbose (INFO), --quiet (FATAL) and --debug (DEBUG)
+    #
+    def self._init_logger(options)
+      # the log log should roll over weekly
+      if options.log_file.nil?
+        if options.debug 
+          # if debug is on but no logfile is specified then log to console
+          options.log_file = STDOUT
+        else
+          # Setting the log level to UNKNOWN effectively turns logging off
+          options.log_level = Logger::UNKNOWN
+        end
+      end
+      options.logger = Logger.new(options.log_file, 'weekly') # roll over weekly
+      options.logger.level = options.log_level
+    end
+   
+    # Takes a string of arguments and splits it into an array, allowing for 'single quotes'
+    # to join words into a single argument.
+    # (Note that parentheses are used to group to exclude the single quotes themselves, but grouping
+    #  results in scan creating an array of arrays with some nil elements hence flatten and delete)
+    def self.sub_args_to_array(subargs)
+      return nil if subargs.nil?
+      return subargs.scan(/([^\s']+)|'([^']*)'/).flatten.delete_if { |x| x.nil?}
+    end
+
+  end #class CLI
+end