finmodeling 0.1 → 0.2

data/Rakefile

@@ -1,6 +1,18 @@
 #!/usr/bin/env rake
 require "bundler/gem_tasks"
 
+desc "run the specs"
 task :test do
+  sh "if git status | grep 'modified:' | grep annual_report >/dev/null; then echo \"\n\nYou should get rid of ~/.finmodeling\"; fi"
   sh "rspec -c -fd -I. -Ispec spec/*spec.rb"
 end
+
+desc "purges anything cached"
+task :purge_cache do
+  sh "rm -rf ~/.finmodeling"
+end
+
+desc "purges anything cached, except the raw XBRL filing downloads."
+task :purge_cache_except_filings do
+  sh "rm -rf ~/.finmodeling/classifiers/ ~/.finmodeling/constructors/ ~/.finmodeling/summaries/"
+end

data/TODO.txt

@@ -1,36 +1,129 @@
-Statement of Shareholder Equity
-	- Parse this one to get at comprehensive income
-
-Forecasts
-	- Choosing of policies should probably be bumped to a higher-level package.
-	- Need more sophisticated forecasting policies
-	- Need to have more sophisticated policies fall back on simpler ones (or none) when there's
-	  too much chaos
-
-Bugs
-	- Not all disclosures *begin* with disclosure.  See CRM's most recent 10-K, for instance.
-	- The "valid vals" thing should hides gaps in data, which silently&crappily affect the regressions.
-	- show_reports doesn't check filing validity
+XBRL items can include Footnotes. Should probably print these out.
 
 Income Statement
-	- Dig into the disclosures. They've got lots more interesting goodness.
-	- It'd be interesting to try to correct for cyclicality.
+	- In IncomeStatement::latest_quarterly_refactored, it's checking whether the statement has any
+	  quarterly periods, and if so, using the last one. There's a problem with that, some statements
+	  do have quarterly periods, but don't break all lines down to the quarterly level. In this example:
+		./examples/show_reports.rb --income-detail CHRW 2011-09-01
+	  The 2012-12-31 column is missing a $283M income (FIBT) because that item is only given in annual
+	  periods (not the 2012-10-01..2012-12-31 quarterly period). See it in this dump:
+		./examples/dump_latest_10k.rb http://www.sec.gov/Archives/edgar/data/1043277/000104327713000004/0001043277-13-000004-index.htm
+	  Need to modify that method to check whether there are quarterly PERIODS and that all leaf items
+	  have that quarterly period.
+	- Improve the readability of the error checking. One option would be to move to
+	  a system like ActiveRecord's error-handling, in which a class just accumulates its errors 
+	  nicely, and lets you print them out whenever it's convenient, and in a well-formatted way, 
+	  without causing an immediate exception.
 	- The classifier that had been classifying unknown credits/debits is disabled. Replace it?
-	- net income != comprehensive income.  pull in items from SSE
+	  (I don't even remember what this refers to...)
+	- net income != comprehensive income.  pull in items from SSE? and/or parse the comprehensive
+	  income statement.
 	- certain things are being misclassified as FIBT instead of OOIBT. (hedging, 
 	  currency exchg, etc)
-	- naming consistency (e.g., rename OIBT as OOIBT?)
-	- consistency between operating/financing vs. operational/financial?
 	- need to better understand minority interest, noncontrollable blah-blah,
-	  etc.
+	  etc. (not a big deal for tech companies, but important for conglomerates, mediaco's, etc)
+	- naming consistency (e.g., rename OIBT as OOIBT?)
+	- naming consistency between operating/financing vs. operational/financial?
+	- Instead of having find_calculation_arc use a discrete set of exact regexes, what about using
+	  a classifier (and minimum confidence threshold), like the row classifier works?
+	- How to tell if a stock gave off a dividend? Are we tracking those somewhere?
+
+Forecasts
+	- Refactor it. The logic is too spread around.
+	- Detect historicals that are too noisy to forecast from:
+		- R^2 of regressions?
+		- Certain ratios remaining constant-ish (e.g, composition ratios)
+	- Detect historicals with poor earnings quality:
+		- Noisy NI/C
+	- Detect forecasts that look suspect:
+		- Derived values in the forecasts (e.g., composition ratio) remain 
+		  with a standard deviation or two from their trend lines.
+	- Allow assumptions to be tweaked by user: 
+		- How is continuing value calculated? (What long-term g?)
+
+Comprehensive Income Statement
+	- Not very many examples, especially of less common things like FIAT. Find more.
+	- The pension and stock-expense OCIs are getting labeled as operating OCI, which is geting
+	  added back in as other operating income after tax. Really, though, they should be treated
+	  as a core operating expense, since they're about paying the core employees.
 
 Balance Sheet
 	- Dig into the disclosures. They've got lots more interesting goodness.
+	- Need to reclassify: credits to shareholders' equity for stock compensation expense
+	- Minority interest?
+	- Need to reclassify: dividends payable
 	- The classifier that had been classifying unknown credits/debits is disabled. Replace it?
+	- "Redeemable Noncontrolling Interest Equity Carrying Amount"??
+
+Try to figure out why the change in equity is not equal to comprehensive income, in cases where we know it.
+	- Some 10k's (and even some 10q's?) have Statements of SE
+	- When they don't:
+		- Some 10q's have AOCI broken out on BS. We can look for deltas
+		- Some 10q's have disclosures that list components of OCI or CI
+		- The remaining residual is probably stock-based compensation expense
+	- minority interests makes it screwey. also, preferred dividends?
 
 Cash Flow Statement
-	- Dig into the disclosures. They've got lots more interesting goodness.
 	- not taking into account: net cash interest, tax on net interest, or non-cash
 	  transactions.  All 3 of these items are listed in supplementary disclosures.
 	- the classifier's success rate isn't that high.
 	- "I" is defined inversely.
+
+Statement of Shareholder Equity
+	- Not taking into account share-based compensation, dividends payable, etc.
+
+Cost of Capital
+	- Debt cost of capital - try to estimate before tax cost of capital by modeling
+	  credit spread as a function of interest coverage. Is there a way to estimate
+	  marginal (not effective) tax rate?
+
+XBRL includes a "weight" parameter. Is that useable in place of the "mapping" idea used here?
+
+########################################################################################################################
+########################################################################################################################
+########################################################################################################################
+
+require 'finmodeling'
+
+report_paths=["/Users/jimlindstrom/.finmodeling/filings/000110465910063219", "/Users/jimlindstrom/.finmodeling/filings/000119312509153165", "/Users/jimlindstrom/.finmodeling/filings/000119312509214859", "/Users/jimlindstrom/.finmodeling/filings/000119312510012085", "/Users/jimlindstrom/.finmodeling/filings/000119312510030774", "/Users/jimlindstrom/.finmodeling/filings/000119312510088957", "/Users/jimlindstrom/.finmodeling/filings/000119312510162840", "/Users/jimlindstrom/.finmodeling/filings/000119312510238044", "/Users/jimlindstrom/.finmodeling/filings/000119312511010144", "/Users/jimlindstrom/.finmodeling/filings/000119312511104388", "/Users/jimlindstrom/.finmodeling/filings/000119312511134428", "/Users/jimlindstrom/.finmodeling/filings/000119312511192493", "/Users/jimlindstrom/.finmodeling/filings/000119312511199078", "/Users/jimlindstrom/.finmodeling/filings/000119312511282113", "/Users/jimlindstrom/.finmodeling/filings/000119312511282235", "/Users/jimlindstrom/.finmodeling/filings/000119312512023398", "/Users/jimlindstrom/.finmodeling/filings/000119312512025336", "/Users/jimlindstrom/.finmodeling/filings/000119312512182321", "/Users/jimlindstrom/.finmodeling/filings/000119312512314552", "/Users/jimlindstrom/.finmodeling/filings/000119312512444068", "/Users/jimlindstrom/.finmodeling/filings/000119312513022339"]
+ 
+report_paths.each do |report_path|
+
+  puts report_path
+
+  filing=FinModeling::AnnualReportFiling.new(report_path)
+  bs=filing.balance_sheet
+  le=bs.liabs_and_equity_calculation
+  le.periods.each do |period|
+    puts le.calculation.leaf_items(period).select{ |x| x.footnotes }.inspect
+  end
+  a=bs.assets_calculation
+  a.periods.each do |period|
+    puts a.calculation.leaf_items(period).select{ |x| x.footnotes }.inspect
+  end
+
+  puts 
+
+end
+#  #<Xbrlware::Item:0x007fcd631b7b18 
+#    @ins=#<Xbrlware::Instance:0x007fcd63b8cdf0>, 
+#    @name="CommonStockValue", 
+#    @context=Id [eol_PE2035----0910-Q0009_STD_0_20091226_0], 
+#    Entity { Identifier { schema [http://www.sec.gov/CIK], value [] }  },  
+#    period [2009-12-26], 
+#    @precision=nil, 
+#    @decimals="-6", 
+#    @footnotes={"en-US"=>["See Note 2, \"Retrospective Adoption of New Accounting Principles\" of this Form 10-Q."]}, 
+#    @value="8962000000.0", 
+#    @unit=#<Xbrlware::Unit:0x007fcd66db28e8 @id="iso4217_USD", @measure=["iso4217:USD"], @ns="http://www.xbrl.org/2003/instance", @nsp="">, 
+#    @ns="http://xbrl.us/us-gaap/2009-01-31", 
+#    @nsp="us-gaap", 
+#    @def={"xbrli:balance"=>"credit", "name"=>"CommonStockValue", "nillable"=>"true", "xbrli:periodType"=>"instant", "id"=>"us-gaap_CommonStockValue", "type"=>"xbrli:monetaryItemType", "substitutionGroup"=>"xbrli:item"}>
+
+
+# several TODOs:
+# 1. modify xbrlware-extras to write these to the constructors
+#    item = Xbrlware::Item.new(instance, name, context, value, unit, precision, decimals, footnotes)
+# 2. modify CalculationSummary to handle printing footnoes
+# 3. modify the various calculations to extract footnoes and stuff them into the summaries.
+# --- And do it TDD-style.

data/examples/{dump_report.rb → dump_latest_10k.rb}

@@ -3,7 +3,7 @@
 require 'finmodeling'
 
 if ARGV.length != 1
-  puts "usage #{__FILE__} <stock symbol>"
+  puts "usage #{__FILE__} <stock symbol | URL>"
   exit
 end
 

data/examples/list_disclosures.rb

@@ -0,0 +1,50 @@
+#!/usr/bin/env ruby
+
+require 'finmodeling'
+
+class Arguments
+  def self.show_usage_and_exit
+    puts "usage:"
+    puts "\t#{__FILE__} <stock symbol> <start date, e.g. '2010-01-01'>"
+    exit
+  end
+  
+  def self.parse(args)
+    a = { :stock_symbol => nil, :start_date => nil }
+
+    self.show_usage_and_exit if args.length != 2
+    a[:stock_symbol]  = args[0]
+    a[:start_date] = Time.parse(args[1])
+  
+    return a
+  end
+end
+
+args = Arguments.parse(ARGV)
+
+company = FinModeling::Company.find(args[:stock_symbol])
+raise RuntimeError.new("couldn't find company") if !company
+puts "company name: #{company.name}"
+
+filings = FinModeling::CompanyFilings.new(company.filings_since_date(args[:start_date]))
+if filings.empty?
+  puts "No filings..."
+  exit
+end
+
+disclosure_periods = {}
+
+filings.each do |filing|
+
+  filing.disclosures.each do |disclosure|
+    disclosure_label = disclosure.summary(:period => disclosure.periods.last).title.gsub(/ \(.*/,'')
+
+    disclosure_periods[disclosure_label] ||= []
+    disclosure_periods[disclosure_label] += disclosure.periods
+  end
+end
+
+disclosure_periods.keys.sort.each do |disclosure_label|
+  puts disclosure_label.to_s + ": " + disclosure_periods[disclosure_label].map{ |x| x.to_pretty_s }.sort.uniq.join(', ')
+end
+

data/examples/show_report.rb

@@ -42,7 +42,7 @@ class Arguments
 
   protected
 
-  def self.parse_just_a_url(raw_args, parsed)
+  def self.parse_just_a_url(raw_args, parsed_args)
     self.show_usage_and_exit if raw_args.length != 2
 
     parsed_args[:filing_url]  = raw_args[0]
@@ -124,21 +124,63 @@ def print_reformulated_balance_sheet(filing, report_type)
 end
 
 def print_income_statement(filing, report_type)
-  period  = filing.income_statement.net_income_calculation.periods.yearly.last    if report_type == :annual_report
-  period  = filing.income_statement.net_income_calculation.periods.quarterly.last if report_type == :quarterly_report
-  puts "Income Statement (#{period.to_pretty_s})"
-
-  summaries = []
-  summaries << filing.income_statement.net_income_calculation.summary(:period => period)
+  if filing.has_an_income_statement?
+    period  = filing.income_statement.net_income_calculation.periods.yearly.last    if report_type == :annual_report
+    period  = filing.income_statement.net_income_calculation.periods.quarterly.last if report_type == :quarterly_report
+    puts "Income Statement (#{period.to_pretty_s})"
+  
+    summaries = []
+    summaries << filing.income_statement.net_income_calculation.summary(:period => period)
+  
+    print_summaries(summaries)
+  else
+    puts "Filing has no income statement."
+    puts 
+  end
+end
 
-  print_summaries(summaries)
+def print_comprehensive_income_statement(filing, report_type)
+  if filing.has_a_comprehensive_income_statement?
+    period  = filing.comprehensive_income_statement.comprehensive_income_calculation.periods.yearly.last    if report_type == :annual_report
+    period  = filing.comprehensive_income_statement.comprehensive_income_calculation.periods.quarterly.last if report_type == :quarterly_report
+    puts "Comprehensive Income Statement (#{period.to_pretty_s})"
+  
+    summaries = []
+    summaries << filing.comprehensive_income_statement.comprehensive_income_calculation.summary(:period => period)
+    #summaries << filing.comprehensive_income_statement.summary(:period => period) # when debugging, try printing the whole statement like this.
+  
+    print_summaries(summaries)
+  else
+    puts "Filing has no comprehensive income statement."
+    puts 
+  end
 end
 
 def print_reformulated_income_statement(filing, report_type)
-  period  = filing.income_statement.net_income_calculation.periods.yearly.last    if report_type == :annual_report
-  period  = filing.income_statement.net_income_calculation.periods.quarterly.last if report_type == :quarterly_report
+  reformed_inc_stmt = nil
+  comprehensive_inc_calc = nil
+
+  if filing.has_a_comprehensive_income_statement?
+    comprehensive_inc_calc = filing.comprehensive_income_statement.comprehensive_income_calculation
+  end
 
-  reformed_inc_stmt  = filing.income_statement.reformulated(period)
+  if filing.has_an_income_statement?
+    period  = filing.income_statement.net_income_calculation.periods.yearly.last    if report_type == :annual_report
+    period  = filing.income_statement.net_income_calculation.periods.quarterly.last if report_type == :quarterly_report
+  
+    reformed_inc_stmt  = filing.income_statement.reformulated(period, comprehensive_inc_calc)
+
+  elsif filing.has_a_comprehensive_income_statement? && 
+        filing.comprehensive_income_statement
+              .comprehensive_income_calculation
+              .has_revenue_item?
+    period  = filing.comprehensive_income_statement.comprehensive_income_calculation.periods.yearly.last    if report_type == :annual_report
+    period  = filing.comprehensive_income_statement.comprehensive_income_calculation.periods.quarterly.last if report_type == :quarterly_report
+  
+    reformed_inc_stmt  = filing.comprehensive_income_statement.reformulated(period, comprehensive_inc_calc)
+  else
+    raise RuntimeError.new("Can't create reformulated income statement")
+  end
 
   summaries = []
   summaries << reformed_inc_stmt.gross_revenue
@@ -154,27 +196,62 @@ end
 def print_cash_flow_statement(filing, report_type)
   period = filing.cash_flow_statement.periods.yearly.last    if report_type == :annual_report
   period = filing.cash_flow_statement.periods.quarterly.last if report_type == :quarterly_report
-  puts "Cash Flow Statement (#{period.to_pretty_s})"
-  
-  summaries = []
-  summaries << filing.cash_flow_statement.cash_change_calculation.summary(:period => period)
 
-  print_summaries(summaries)
+  if period
+    puts "cash flow statement (#{period.to_pretty_s})"
+    
+    summaries = []
+    summaries << filing.cash_flow_statement.cash_change_calculation.summary(:period => period)
+  
+    print_summaries(summaries)
+  else
+    puts "WARNING: cash flow statement period is nil!"
+  end
 end
 
 def print_reformulated_cash_flow_statement(filing, report_type)
   period = filing.cash_flow_statement.periods.yearly.last    if report_type == :annual_report
   period = filing.cash_flow_statement.periods.quarterly.last if report_type == :quarterly_report
+
+  if period
+    reformed_cash_flow_stmt  = filing.cash_flow_statement.reformulated(period)
+  
+    summaries = []
+    summaries << reformed_cash_flow_stmt.cash_from_operations
+    summaries << reformed_cash_flow_stmt.cash_investments_in_operations
+    summaries << reformed_cash_flow_stmt.payments_to_debtholders
+    summaries << reformed_cash_flow_stmt.payments_to_stockholders
+    summaries << reformed_cash_flow_stmt.free_cash_flow
+    summaries << reformed_cash_flow_stmt.financing_flows
+  
+    print_summaries(summaries)
+  else
+    puts "WARNING: reformulated cash flow statement period is nil!"
+  end
+end
+
+def print_shareholder_equity_statement(filing, report_type)
+  return if !filing.has_a_shareholder_equity_statement?
+  period = filing.shareholder_equity_statement.periods.yearly.last    if report_type == :annual_report
+  period = filing.shareholder_equity_statement.periods.quarterly.last if report_type == :quarterly_report
+  puts "shareholder equity statement (#{period.to_pretty_s})"
+  
+  summaries = []
+  summaries << filing.shareholder_equity_statement.equity_change_calculation.summary(:period => period)
+
+  print_summaries(summaries)
+end
+
+def print_reformulated_shareholder_equity_statement(filing, report_type)
+  return if !filing.has_a_shareholder_equity_statement?
+  period = filing.shareholder_equity_statement.periods.yearly.last    if report_type == :annual_report
+  period = filing.shareholder_equity_statement.periods.quarterly.last if report_type == :quarterly_report
   
-  reformed_cash_flow_stmt  = filing.cash_flow_statement.reformulated(period)
+  reformed_shareholder_equity_stmt = filing.shareholder_equity_statement.reformulated(period)
 
   summaries = []
-  summaries << reformed_cash_flow_stmt.cash_from_operations
-  summaries << reformed_cash_flow_stmt.cash_investments_in_operations
-  summaries << reformed_cash_flow_stmt.payments_to_debtholders
-  summaries << reformed_cash_flow_stmt.payments_to_stockholders
-  summaries << reformed_cash_flow_stmt.free_cash_flow
-  summaries << reformed_cash_flow_stmt.financing_flows
+  summaries << reformed_shareholder_equity_stmt.transactions_with_shareholders
+  summaries << reformed_shareholder_equity_stmt.comprehensive_income
 
   print_summaries(summaries)
 end
@@ -205,14 +282,17 @@ if args[:filing_url].nil?
   args[:filing_url] = get_company_filing_url(args[:stock_symbol], args[:report_type], args[:report_offset])
 end
 
-filing = get_filing(        args[:filing_url], args[:report_type])
-
-print_balance_sheet(                   filing, args[:report_type])
-print_reformulated_balance_sheet(      filing, args[:report_type])
-print_income_statement(                filing, args[:report_type])
-print_reformulated_income_statement(   filing, args[:report_type])
-print_cash_flow_statement(             filing, args[:report_type])
-print_reformulated_cash_flow_statement(filing, args[:report_type])
-print_disclosures(                     filing, args[:report_type]) if args[:show_disclosures]
+filing = get_filing(args[:filing_url], args[:report_type])
+
+print_balance_sheet(                            filing, args[:report_type])
+print_reformulated_balance_sheet(               filing, args[:report_type])
+print_income_statement(                         filing, args[:report_type])
+print_comprehensive_income_statement(           filing, args[:report_type])
+print_reformulated_income_statement(            filing, args[:report_type])
+print_cash_flow_statement(                      filing, args[:report_type])
+print_reformulated_cash_flow_statement(         filing, args[:report_type])
+print_shareholder_equity_statement(             filing, args[:report_type])
+print_reformulated_shareholder_equity_statement(filing, args[:report_type])
+print_disclosures(                              filing, args[:report_type]) if args[:show_disclosures]
 
 raise RuntimeError.new("filing is not valid") if !filing.is_valid?

data/examples/show_reports.rb

@@ -9,42 +9,108 @@ class Arguments
     puts
     puts "\tOptions:"
     puts "\t\t--num-forecasts <num>: how many periods to forecast"
+    puts "\t\t  --forecast-policy <trailing_avg|linear_trend> (default: linear_trend)"
+    puts "\t\t  --do-valuation: value the company's equity. (requires >= 2 forecasts)"
+    puts "\t\t    --marginal-tax-rate <num>: default is 0.36 (36%)"
+    puts "\t\t    --before-tax-cost-of-debt <num>: default is 0.05 (5%)"
     puts "\t\t--no-cache: disable caching"
     puts "\t\t--balance-detail: show details about the balance sheet calculation"
     puts "\t\t--income-detail: show details about the net income calculation"
+    puts "\t\t--show-disclosure <part of title>: show a particular disclosure over time"
+    puts "\t\t--show-regressions: show the regressions of calculations that are used to do forecasts"
     exit
   end
-  
-  def self.parse(args)
-    a = { :stock_symbol => nil, :start_date => nil, :num_forecasts => nil }
-
-    while args.any? && args.first =~ /^--/
-      case args.first.downcase
-        when '--no-cache'
-          FinModeling::Config.disable_caching
-          puts "Caching is #{FinModeling::Config.caching_enabled? ? "enabled" : "disabled"}"
-        when '--balance-detail'
-          FinModeling::Config.enable_balance_detail
-          puts "Balance sheet detail is #{FinModeling::Config.balance_detail_enabled? ? "enabled" : "disabled"}"
-        when '--income-detail'
-          FinModeling::Config.enable_income_detail
-          puts "Net income detail is #{FinModeling::Config.income_detail_enabled? ? "enabled" : "disabled"}"
-        when '--num-forecasts'
-          a[:num_forecasts] = args[1].to_i
-          self.show_usage_and_exit unless a[:num_forecasts] >= 1
-          puts "Forecasting #{a[:num_forecasts]} periods"
-          args = args[1..-1]
-        else
-          self.show_usage_and_exit
-      end
-      args = args[1..-1]
+   
+  def self.parse(raw_args)
+    parsed_args = self.default_options
+
+    while raw_args.any? && raw_args.first =~ /^--/
+      self.parse_next_option(raw_args, parsed_args)
     end
   
-    self.show_usage_and_exit if args.length != 2
-    a[:stock_symbol]  = args[0]
-    a[:start_date] = Time.parse(args[1])
+    self.show_usage_and_exit if raw_args.length != 2
+    parsed_args[:stock_symbol] = raw_args[0]
+    parsed_args[:start_date  ] = Time.parse(raw_args[1])
   
-    return a
+    return parsed_args
+  end
+
+  private
+
+  def self.default_options
+    { :stock_symbol            => nil, 
+      :start_date              => nil, 
+      :num_forecasts           => nil, 
+      :do_valuation            => false, 
+      :forecast_policy         => :linear_trend,
+      :marginal_tax_rate       => 0.36, 
+      :before_tax_cost_of_debt => 0.05, 
+      :show_regressions        => false, 
+      :disclosures             => [ ] }
+  end
+
+
+  def self.parse_next_option(raw_args, parsed_args)
+    case raw_args.first.downcase
+      when '--no-cache'
+        FinModeling::Config.disable_caching
+        puts "Caching is #{FinModeling::Config.caching_enabled? ? "enabled" : "disabled"}"
+
+      when '--balance-detail'
+        FinModeling::Config.enable_balance_detail
+        puts "Balance sheet detail is #{FinModeling::Config.balance_detail_enabled? ? "enabled" : "disabled"}"
+
+      when '--income-detail'
+        FinModeling::Config.enable_income_detail
+        puts "Net income detail is #{FinModeling::Config.income_detail_enabled? ? "enabled" : "disabled"}"
+
+      when '--num-forecasts'
+        self.show_usage_and_exit if raw_args.length < 2
+        parsed_args[:num_forecasts] = raw_args[1].to_i
+        self.show_usage_and_exit unless parsed_args[:num_forecasts] >= 1
+        puts "Forecasting #{parsed_args[:num_forecasts]} periods"
+        raw_args.shift
+
+      when '--do-valuation'
+        parsed_args[:do_valuation] = true
+        puts "Doing valuation"
+
+      when '--forecast-policy'
+        self.show_usage_and_exit if raw_args.length < 2
+        parsed_args[:forecast_policy] = raw_args[1].to_sym
+        self.show_usage_and_exit unless [:trailing_avg, :linear_trend].include?(parsed_args[:forecast_policy])
+        puts "Forecast policy: #{parsed_args[:forecast_policy]}"
+        raw_args.shift
+
+      when '--before-tax-cost-of-debt'
+        self.show_usage_and_exit if raw_args.length < 2
+        parsed_args[:before_tax_cost_of_debt] = raw_args[1].to_f
+        self.show_usage_and_exit unless parsed_args[:before_tax_cost_of_debt] >= 0.00 && parsed_args[:before_tax_cost_of_debt] <= 1.00
+        puts "before tax cost of debt: #{parsed_args[:before_tax_cost_of_debt]}"
+        raw_args.shift
+
+      when '--marginal-tax-rate'
+        self.show_usage_and_exit if raw_args.length < 2
+        parsed_args[:marginal_tax_rate] = raw_args[1].to_f
+        self.show_usage_and_exit unless parsed_args[:marginal_tax_rate] >= 0.00 && parsed_args[:marginal_tax_rate] <= 1.00
+        puts "marginal tax rate: #{parsed_args[:marginal_tax_rate]}"
+        raw_args.shift
+
+      when '--show-regressions'
+        parsed_args[:show_regressions] = true
+        puts "Showing regressions"
+
+      when '--show-disclosure'
+        self.show_usage_and_exit if raw_args.length < 2
+        parsed_args[:disclosures] << raw_args[1]
+        puts "Showing disclosure: #{parsed_args[:disclosures].last}"
+        raw_args.shift
+
+      else
+        self.show_usage_and_exit
+
+    end
+    raw_args.shift
   end
 end
 
@@ -59,19 +125,82 @@ if filings.empty?
   puts "No filings..."
   exit
 end
+filings.each do |filing|
+  begin
+    if !filing.is_valid?
+      raise RuntimeError.new("filing is not valid. type: #{filing.class}. period: #{filing.balance_sheet.periods.last}.") 
+    end
+  rescue FinModeling::InvalidFilingError => e
+    pre_msg = "\n\nFiling is not valid. type: #{filing.class}. period: #{filing.balance_sheet.periods.last}.\n"
+    raise e, pre_msg+e.message, e.backtrace
+  end
+end
+
+
+fl  = filings.re_bs_arr.last.financial_liabilities.total
+dcoc = FinModeling::DebtCostOfCapital.calculate(:before_tax_cost   => FinModeling::Rate.new(args[:before_tax_cost_of_debt]), 
+                                                :marginal_tax_rate => FinModeling::Rate.new(args[:marginal_tax_rate]))
+ecoc = FinModeling::CAPM::EquityCostOfCapital.from_ticker(args[:stock_symbol])
+#ecoc = FinModeling::FamaFrench::EquityCostOfCapital.from_ticker(args[:stock_symbol])
+if (ecoc.value < 0.05) || (ecoc.value > 0.30)
+  puts "WARNING: cost of equity capital is highly suspect..."
+end
+wacc = FinModeling::WeightedAvgCostOfCapital.new(equity_market_val      = YahooFinance::get_market_cap(args[:stock_symbol].dup),
+                                                 debt_market_val        = fl,
+                                                 cost_of_equity         = ecoc,
+                                                 after_tax_cost_of_debt = dcoc)
 
-forecasts = filings.forecasts(filings.choose_forecasting_policy, num_quarters=args[:num_forecasts]) if args[:num_forecasts]
+forecasts = filings.forecasts(filings.choose_forecasting_policy(wacc.rate.value, args[:forecast_policy]), num_quarters=args[:num_forecasts]) if args[:num_forecasts]
 
 bs_analyses = filings.balance_sheet_analyses 
 bs_analyses += forecasts.balance_sheet_analyses(filings) if forecasts
 bs_analyses.totals_row_enabled = false 
 bs_analyses.print
-filings.balance_sheet_analyses.print_extras if filings.balance_sheet_analyses.respond_to?(:print_extras)
+if args[:show_regressions] && filings.balance_sheet_analyses.respond_to?(:print_regressions)
+  filings.balance_sheet_analyses.print_regressions 
+end
 
-is_analyses = filings.income_statement_analyses 
-is_analyses += forecasts.income_statement_analyses(filings) if forecasts
+is_analyses = filings.income_statement_analyses(wacc.rate.value)
+is_analyses += forecasts.income_statement_analyses(filings, wacc.rate.value) if forecasts
 is_analyses.totals_row_enabled = false 
 is_analyses.print
-filings.income_statement_analyses.print_extras if filings.income_statement_analyses.respond_to?(:print_extras)
+if args[:show_regressions] && filings.income_statement_analyses.respond_to?(:print_regressions)
+  filings.income_statement_analyses.print_regressions
+end
 
 filings.cash_flow_statement_analyses.print
+if args[:show_regressions] && filings.cash_flow_statement_analyses.respond_to?(:print_regressions)
+  filings.cash_flow_statement_analyses.print_regressions
+end
+
+args[:disclosures].each do |disclosure_title|
+  title_regex = Regexp.new(disclosure_title, Regexp::IGNORECASE)
+  disclosures   = filings.disclosures(title_regex, :quarterly)
+  disclosures ||= filings.disclosures(title_regex, :yearly   )
+  disclosures ||= filings.disclosures(title_regex            )
+  if disclosures
+    disclosures.auto_scale!
+    if (disclosures.header_row.vals - bs_analyses.header_row.vals).length == 0
+      0.upto(bs_analyses.header_row.vals.length-1) do |idx|
+        if bs_analyses.header_row.vals[idx] != disclosures.header_row.vals[idx]
+          disclosures.insert_column_before(idx, "")
+        end
+      end
+    end
+    disclosures.print
+  else
+    puts "Couldn't find disclosures called: #{title_regex}"
+  end
+end
+
+if args[:do_valuation]
+  if args[:num_forecasts] && args[:num_forecasts]>=2
+    wacc.summary.print 
+
+    num_shares = YahooFinance::get_num_shares(args[:stock_symbol].dup)
+    valuation = FinModeling::ReOIValuation.new(filings, forecasts, wacc.rate, num_shares)
+    valuation.summary.print
+  else
+    puts "Oops. Can't do valuation without >= 2 forecasts"
+  end
+end

data/finmodeling.gemspec

@@ -13,11 +13,14 @@ Gem::Specification.new do |gem|
   gem.add_dependency("edgar")
 
   gem.add_dependency("xbrlware-ruby19", "1.1.2.19.2")
-  gem.add_dependency("xbrlware-extras", "1.1.2.19.2")
+  gem.add_dependency("xbrlware-extras", "1.1.2.19.3")
+  gem.add_dependency("nasdaq_query")
 
   gem.add_dependency("sec_query")
   gem.add_dependency("naive_bayes")
   gem.add_dependency("statsample")
+  gem.add_dependency("yahoofinance")
+  gem.add_dependency("gsl")
 
   gem.add_development_dependency("rspec", "2.5")
   gem.add_development_dependency("rake")