sqa 0.0.15 → 0.0.18

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0fcc7dbda5b62549fa1fb9691bf7ef9af1a0ba5dfcff63494ce2cdf6ca7e578a
-  data.tar.gz: 9527508fa1573a09536ef4bf735671761d722d1611494d3aeb8615ba25385f2c
+  metadata.gz: b7733f5cb84c4ccbad5e68105e818b35f1aca450fd97eeb3186cfb9a1962d44e
+  data.tar.gz: a9b2bf2a0fa6a91de1367fd73beac6a28d263024ded37c216520120cc3181cc9
 SHA512:
-  metadata.gz: c5ce2164cd95cf91c62e054dad4fef84ed105d7713207bf5f5e39fbad282785184217ed303fe73b24801b871e9ca0db371c0f514949803aace37b788ebec6140
-  data.tar.gz: f7336d71e365388d571267e87fbdd82a66dbd532da8e386e9670e1a9f7034d2279202e6df16ce30d14410b2ab39923ecf4bb78e0941b554bf11ee3101fdb9a41
+  metadata.gz: 27280c675c37a92f450d5108c40848cc65a3975e8e783ca25cbbee0bfa18b1c570d697c4c33d4fd73d84c57d717df3e6cbb8667c51fb3ed8fe48df509f9d1f7e
+  data.tar.gz: 475333a102814f5bef21dbea61126bf6b0b61836cccb6a333acc2dbcaa2ba5bc14c8c9191b31065777ca984fcec883fb37da36b07a9121f2afeb0a0004bc841b

data/README.md

@@ -1,18 +1,13 @@
 # SQA - Simple Qualitative Analysis
 
-This is a very simplistic set of tools for running technical analysis on a stock portfolio.  Simplistic means it is not reliable nor intended for any kind of financial use.  Think of it as a training tool.  I do.  Its helping me understand why I need professional help from people who know what they are doing.
+This is a very simplistic set of tools for running technical analysis (quantitative and qualitative) on a stock portfolio.  Simplistic means it is not reliable nor intended for any kind of mission-critical financial use.  Think of it as a training tool.  I do.  Its helping me understand why I need professional help from people who know what they are doing.
 
-The BUY/SELL signals that it generates are part of a game.  **DO NOT USE** when real money is at stake.
+The BUY/SELL signals that it generates should not be taken seriously.  **DO NOT USE** this library when real money is at stake.  If you lose your shirt playing in the stock market don't come crying to me.  I think playing in the market is like playing in the street.  You are going to get run over.
 
 ## This is a Work in Progress
 
-I'm making use of lots of gems which may not be part of the gemspec at this time.  I will be adding them as they make the final cut as to fitness for the intended function.  Some gems are configurable.  For example the default for the plotting library is `gruff`.  There are several available that the `daru` gem can use.
+I am experimenting with different gems to support various functionality.  Sometimes they do not work out well.  For example I've gone through two different gems to implement the data frame capability.  Neither did what I wanted so I ended up creating my own data frame class based upon the old tried and true Hashie library.
 
-### DARU or RedAmber
-
-I'm just really using `daru` for its data frame object;  However, I just learned about the RedAmber data frame object in Ruby based off of Apache Arrow.  I'm going to look at that product since it is actively maintained.
-
-https://github.com/red-data-tools/red_amber
 
 ## Installation
 
@@ -24,62 +19,200 @@ If bundler is not being used to manage dependencies, install the gem by executin
 
     gem install sqa
 
-## ShoutOut To `daru`
+### Semantic Versioning
 
-**D**ata **A**nalysis in **RU**by
+```ruby
+SQA.version # returns SemVersion object
+exit(1) unless SQA.version >= SemVersion("1.0.0")
+```
 
-http://github.com/v0dro/daru
+## Usage
 
-Its `DataFrame` class is a very interesting in memory data structure.
+**Do not use!** but its okay to play with.
 
-## Usage
+`SQA` can be used from the command line or as a library in your own application.
+
+`SQA` has a command line component.
+
+```plaintext
+$ sqa --help
+Stock Quantitative Analysis (SQA)
+
+Usage: sqa [analysis|web] [OPTIONS]
+
+A collection of things
+
+Options:
+  -c, --config string             Path to the config file
+      --data-dir string           Set the directory for the SQA data
+  -d, --debug                     Turn on debugging output
+      --dump-config path_to_file  Dump the current configuration
+  -h, --help                      Print usage
+  -l, --log_level string          Set the log level (debug, info, warn, error,
+                                  fatal)
+  -p, --portfolio string          Set the filename of the portfolio
+  -t, --trades string             Set the filename into which trades are
+                                  stored
+  -v, --verbose                   Print verbosely
+      --version                   Print version
+
+Examples:
+  sqa -c ~/.sqa.yml -p portfolio.csv -t trades.csv --data-dir ~/sqa_data
+
+  Optional Command Available:
+    analysis - Provide an Analysis of a Portfolio
+    web - Run a web server
+
+WARNING: This is a toy, a play thing, not intended for serious use.
+```
+### Setup a Config File
+
+You will need to create a directory to store the `sqa` data and a configuration file.  You can start by doing this:
+
+```ruby
+gem install sqa
+mkdir ~/Documents/sqa_data
+sqa --data-dir ~/Documents/sqa_data --dump-config ~/.sqa.yml
+```
+
+By default `SQA` looks for a configuration file named `.sqa.yml` in the current directory.  If it does not find one there it looks in the home directory.  You can use the `--config` CLI option to specify a path to your custom config file name.
+
+
+### AlphaVantage
+
+`SQA` makes use of the `AlphaVantage` API to get some stock related information.  You will need an API key in order to use this functionality.  They have a free rate limited API key which allows 5 accesses per second; total of 100 accesses in a day.  If you are doing more than that you are not playing an ought to purchase one of there serious API key plans.
+
+[https://www.alphavantage.co/](https://www.alphavantage.co/)
 
-**Do not use!**
 
 ## Playing in IRB
 
-You can play around in IRB with the SQA framework in two different areas.  First is the stocks and indicators.  The second is with trading strategies.
+You can play around in IRB with the SQA framework.
+
 
 ### With Stocks and Indicators
 
-You will need some CSV files.
+You will need some CSV files.  If you ask for a stock to which you have not existing historical price data in a CSV file, `SQA` can use either Alpha Vantage or Yahoo Finance to get some data.  I like Alpha Vantage better because it has a well defined and documented API.  Yahoo Finance on the other hand does not.  You can manually download historical stock price data from Yahoo Finance into you `sqa data directory`
+
+Historical price data is kept in the `SQA.data_dir` in a CSV file whose name is all lowercase.  If you download the CSV file for the stock symbol "AAPL" it should be saved in you `SQA.data_dir` as `aapl.csv`
 
 #### Get Historical Prices
 
-Go to https::/finance.yahoo.com and down some historical price data for your favorite stocks.  Put those CSV files in to the `sqa_data` directory in your HOME directory.
+Go to https::/finance.yahoo.com and down some historical price data for your favorite stocks.  Put those CSV files into the `SQA.data_dir`.
 
 You may need to create a `portfolio.csv` file or you may not.  TODO
 
-The CSV files will be named by the stock's ticker symbol.  For example: AAPL.csv
+The CSV files will be named by the stock's ticker symbol.  For example: `aapl.csv`
+
+### Playing in the IRB - Setup
 
 ```ruby
 require 'sqa'
-# TODO: See the documentation on configurable items
-# Omit to use defaults
-SQA::Config.from_file(path_to_config_file)
+require 'sqa/cli'
+
+# You can pass a set of CLI options in a String
+SQA.init "-c ~/.sqa.yml"
+
+aapl = SQA::Stock.new(ticker: 'aapl', source: :alpha_vantage)
+#=> aapl with 1207 data points from 2019-01-02 to 2023-10-17
+```
+
+`aapl.df`   is the data frame.  It is implemented as a Hashie::Mash obect -- a Hash or Arrays.
+
+```ruby
+aapl.df.keys
+#=> [:timestamp, :open_price, :high_price, :low_price, :close_price, :adj_close_price, :volume]
+
+aapl.df.adj_close_price.last(5)
+#=> [179.8, 180.71, 178.85, 178.72, 177.15]
+```
+
+`aapl.data` is basic static data, company name, industry etc.  It is also implemented as a Hassie::Mash object but is primary treated as a plain hash object.
+
+```ruby
+aapl.data.keys
+#=> [:ticker, :source, :indicators, :overview]
+
+aapl.data.source
+=> "alpha_vantage"
 
-# initialize framework from configuration values
-SQA.init
+aapl.data.overciew.keys
+=> [:symbol, :asset_type, :name, :description, :cik, :exchange, :currency, :country, :sector, :industry, :address, :fiscal_year_end, :latest_quarter, :market_capitalization, :ebitda, :pe_ratio, :peg_ratio, :book_value, :dividend_per_share, :dividend_yield, :eps, :revenue_per_share_ttm, :profit_margin, :operating_margin_ttm, :return_on_assets_ttm, :return_on_equity_ttm, :revenue_ttm, :gross_profit_ttm, :diluted_epsttm, :quarterly_earnings_growth_yoy, :quarterly_revenue_growth_yoy, :analyst_target_price, :trailing_pe, :forward_pe, :price_to_sales_ratio_ttm, :price_to_book_ratio, :ev_to_revenue, :ev_to_ebitda, :beta, :"52_week_high", :"52_week_low", :"50_day_moving_average", :"200_day_moving_average", :shares_outstanding, :dividend_date, :ex_dividend_date]
 
-aapl = SQA::Stock.new('aapl')
 ```
 
-`aapl.df` is the Daru::DataFrame
-see the `daru` gem for how to manipulate the DataFrame
+### Playing in the IRB - Statistics
+
+Basic statistics are available on all of the SQA::DataFrame arrays.
+
+```ruby
+require 'amazing_print' # to get the ap command
+
+# Look at some summary stats on the last 5 days of
+# adjusted closing pricess of AAPL
+ap aapl.df.adj_close_price.last(5).summary
+{
+                        :frequencies => {
+         179.8 => 1,
+        180.71 => 1,
+        178.85 => 1,
+        178.72 => 1,
+        177.15 => 1
+    },
+                                :max => 180.71,
+                               :mean => 179.046,
+                             :median => 178.85,
+                           :midrange => 178.93,
+                                :min => 177.15,
+                               :mode => nil,
+                        :proportions => {
+         179.8 => 0.2,
+        180.71 => 0.2,
+        178.85 => 0.2,
+        178.72 => 0.2,
+        177.15 => 0.2
+    },
+                          :quartile1 => 178.85,
+                          :quartile2 => 179.8,
+                          :quartile3 => 180.71,
+                              :range => 3.5600000000000023,
+                               :size => 5,
+                                :sum => 895.23,
+    :sample_coefficient_of_variation => 0.006644656242680533,
+                    :sample_kurtosis => 2.089087404921432,
+                        :sample_size => 5,
+                    :sample_skewness => -0.2163861377512453,
+          :sample_standard_deviation => 1.1896991216269788,
+              :sample_standard_error => 0.532049621745943,
+                    :sample_variance => 1.415384000000005,
+                     :sample_zscores => {
+         179.8 => 0.6337736880639895,
+        180.71 => 1.3986729667618856,
+        178.85 => -0.16474753695031497,
+        178.72 => -0.2740188624785824,
+        177.15 => -1.5936802553969298
+    }
+}
+#=> nil
+```
+
+### Playing in the IRB - Indicators
+
 The SQA::Indicator class methods use Arrays not the DataFrame
 Here is an example:
 
 
 ```ruby
-prices = aapl.df.adj_close_price.to_a
+prices = aapl.df.adj_close_price
 period = 14 # size of the window in prices to analyze
 
 rsi = SQA::Indicator.rsi(prices, period)
+#=> {:rsi=>63.46652828230407, :trend=>:normal}
 ```
 
-### With Strategies
+### Playing in the IRB - Strategies
 
-The strategies work off of an Object that contains the information required to make its recommendation.  Build on the previous Ruby snippet ...
+The strategies work off of an Object that contains the information required to make its recommendation.  Building on the previous Ruby snippet ...
 
 ```ruby
 require 'ostruct'
@@ -96,19 +229,24 @@ ss.add SQA::Strategy::RSI
 
 # This is an Array with each "trade" method
 # that is defined in each strategy added
-ss.strategies
+ap ss.strategies
+[
+    [0] SQA::Strategy::Random#trade(vector),
+    [1] SQA::Strategy::RSI#trade(vector)
+]
 
 # Execute those strategies
 results = ss.execute(vector)
+#=> [:hold, :hold]
 ```
 
 `results` is an Array with an entry for each strategy executed.  The entries are either :buy, :sell or :hold.
 
 Currently the strategies are executed sequentially so the results can easily be mapped back to which strategy produced which result.  In the future that will change so that the strategies are executed concurrently.  When that change is introduced the entries in the `results` object will change -- most likely to an Array of Hashes.
 
-### See my **experiments** Repository in the **stocks** Directory
+Any specific strategy may not work on every stock. Using the historical data, it is possible to see which strategy works better for a specific stock.  **Of course the statistical motto is that historical performance is not a fail-proof indicator for future performance.**
 
-I have a program `analysis.rb` that I'm writing along with this `sqa` gem.  Its intended as a practical example/test case for how the gem can be used to analyze a complete portfolio one stock at a time.
+The strategies that come with the `SQA::Strategy` class are examples only.  Its expected that you will come up with your own.  If you do, consider sharing them.
 
 ## Contributing
 

data/checksums/sqa-0.0.17.gem.sha512

@@ -0,0 +1 @@
+2ee94a54d6ac3d13685dc9b91a2bae0fe75feab6148e1aa9a9d4096961b9b7b577b7ce9d1264f0cce260640515ddd86d5fd5fd2b66f49175844c903581ff6fd9

data/checksums/sqa-0.0.18.gem.sha512

@@ -0,0 +1 @@
+0b3d327017ae67b0ce46082acff8bebdb4a981575e9818418c7279de286a233dde003877f3907b37f3f7e0de236015b4ee4404818d692509c805641862025bc6

data/lib/sqa/data_frame/alpha_vantage.rb

@@ -5,9 +5,8 @@
 #
 
 
-class SQA::DataFrame < Daru::DataFrame
+class SQA::DataFrame
   class AlphaVantage
-    API_KEY     = Nenv.av_api_key
     CONNECTION  = Faraday.new(url: 'https://www.alphavantage.co')
     HEADERS     = YahooFinance::HEADERS
 
@@ -24,47 +23,16 @@ class SQA::DataFrame < Daru::DataFrame
       "volume"          => HEADERS[6]
     }
 
+    TRANSFORMERS  = {
+      HEADERS[1] => -> (v) { v.to_f.round(3) },
+      HEADERS[2] => -> (v) { v.to_f.round(3) },
+      HEADERS[3] => -> (v) { v.to_f.round(3) },
+      HEADERS[4] => -> (v) { v.to_f.round(3) },
+      HEADERS[5] => -> (v) { v.to_f.round(3) },
+      HEADERS[6] => -> (v) { v.to_i }
+    }
 
     ################################################################
-    # Load a Dataframe from a csv file
-    def self.load(ticker, type="csv")
-      filepath = SQA.data_dir + "#{ticker}.#{type}"
-
-      if filepath.exist?
-        df = normalize_vector_names SQA::DataFrame.load(ticker, type)
-      else
-        df = recent(ticker, full: true)
-        df.send("to_#{type}",filepath)
-      end
-
-      df
-    end
-
-
-    # Normalize the vector (aka column) names as
-    # symbols using the standard names set by
-    # Yahoo Finance ... since it was the first one
-    # not because its anything special.
-    #
-    def self.normalize_vector_names(df)
-      headers = df.vectors.to_a
-
-      # convert vector names to symbols
-      # when they are strings.  They become stings
-      # when the data frame is saved to a CSV file
-      # and then loaded back in.
-
-      if headers.first == HEADERS.first.to_s
-        a_hash = {}
-        HEADERS.each {|k| a_hash[k.to_s] = k}
-        df.rename_vectors(a_hash) # renames from String to Symbol
-      else
-        df.rename_vectors(HEADER_MAPPING)
-      end
-
-      df
-    end
-
 
     # Get recent data from JSON API
     #
@@ -82,7 +50,8 @@ class SQA::DataFrame < Daru::DataFrame
     #       and adding that to the data frame as if it were
     #       adjusted.
     #
-    def self.recent(ticker, full: false)
+    def self.recent(ticker, full: false, from_date: nil)
+
       # NOTE: Using the CSV format because the JSON format has
       #       really silly key values.  The column names for the
       #       CSV format are much better.
@@ -90,7 +59,7 @@ class SQA::DataFrame < Daru::DataFrame
         "/query?" +
         "function=TIME_SERIES_DAILY&" +
         "symbol=#{ticker.upcase}&" +
-        "apikey=#{API_KEY}&" +
+        "apikey=#{SQA.av.key}&" +
         "datatype=csv&" +
         "outputsize=#{full ? 'full' : 'compact'}"
       ).to_hash
@@ -100,18 +69,19 @@ class SQA::DataFrame < Daru::DataFrame
       end
 
       raw           = response[:body].split
-
       headers       = raw.shift.split(',')
+
       headers[0]    = 'date'  # website returns "timestamp" but that
                               # has an unintended side-effect when
                               # the names are normalized.
+                              # SMELL: IS THIS STILL TRUE?
 
       close_inx     = headers.size - 2
       adj_close_inx = close_inx + 1
 
       headers.insert(adj_close_inx, 'adjusted_close')
 
-      data    = raw.map do |e|
+      aofh    = raw.map do |e|
                   e2 = e.split(',')
                   e2[1..-2] = e2[1..-2].map(&:to_f) # converting open, high, low, close
                   e2[-1]    = e2[-1].to_i           # converting volumn
@@ -119,35 +89,20 @@ class SQA::DataFrame < Daru::DataFrame
                   headers.zip(e2).to_h
                 end
 
-      # What oldest data first in the data frame
-      normalize_vector_names Daru::DataFrame.new(data.reverse)
-    end
-
+      if from_date
+        aofh.reject!{|e| Date.parse(e['date']) < from_date}
+      end
 
-    # Append update_df rows to the base_df
-    #
-    # base_df is ascending on timestamp
-    # update_df is descending on timestamp
-    #
-    # base_df content came from CSV file downloaded
-    # from Yahoo Finance.
-    #
-    # update_df came from scraping the webpage
-    # at Yahoo Finance for the recent history.
-    #
-    # Returns a combined DataFrame.
-    #
-    def self.append(base_df, updates_df)
-      last_timestamp  = Date.parse base_df.timestamp.last
-      filtered_df     = updates_df.filter_rows { |row| Date.parse(row[:timestamp]) > last_timestamp }
+      return nil if aofh.empty?
 
-      last_inx = filtered_df.size - 1
+      # ensure tha the data frame is
+      # always sorted oldest to newest.
 
-      (0..last_inx).each do |x|
-        base_df.add_row filtered_df.row[last_inx-x]
+      if aofh.first['date'] > aofh.last['date']
+        aofh.reverse!
       end
 
-      base_df
+      SQA::DataFrame.from_aofh(aofh, mapping: HEADER_MAPPING, transformers: TRANSFORMERS)
     end
   end
 end

data/lib/sqa/data_frame/yahoo_finance.rb

@@ -2,7 +2,7 @@
 # frozen_string_literal: true
 
 
-class SQA::DataFrame < Daru::DataFrame
+class SQA::DataFrame
   class YahooFinance
     CONNECTION  = Faraday.new(url: 'https://finance.yahoo.com')
     HEADERS     = [
@@ -30,21 +30,6 @@ class SQA::DataFrame < Daru::DataFrame
       }
 
     ################################################################
-    def self.load(filename, options={}, &block)
-      df = SQA::DataFrame.load(filename, options={}, &block)
-
-      headers = df.vectors
-
-      if headers.first == HEADERS.first.to_s
-        a_hash = {}
-        HEADERS.each {|k| a_hash[k.to_s] = k}
-        df.rename_vectors(a_hash)
-      else
-        df.rename_vectors(HEADER_MAPPING)
-      end
-
-      df
-    end
 
 
     # Scrape the Yahoo Finance website to get recent
@@ -62,7 +47,7 @@ class SQA::DataFrame < Daru::DataFrame
 
       rows      = table.css('tbody tr')
 
-      data = []
+      aofh = []
 
       rows.each do |row|
         cols = row.css('td').map{|c| c.children[0].text}
@@ -80,37 +65,10 @@ class SQA::DataFrame < Daru::DataFrame
         cols[0] = Date.parse(cols[0]).to_s
         cols[6] = cols[6].tr(',','').to_i
         (1..5).each {|x| cols[x] = cols[x].to_f}
-        data << HEADERS.zip(cols).to_h
-      end
-
-      Daru::DataFrame.new(data)
-    end
-
-
-    # Append update_df rows to the base_df
-    #
-    # base_df is ascending on timestamp
-    # update_df is descending on timestamp
-    #
-    # base_df content came from CSV file downloaded
-    # from Yahoo Finance.
-    #
-    # update_df came from scraping the webpage
-    # at Yahoo Finance for the recent history.
-    #
-    # Returns a combined DataFrame.
-    #
-    def self.append(base_df, updates_df)
-      last_timestamp  = Date.parse base_df.timestamp.last
-      filtered_df     = updates_df.filter_rows { |row| Date.parse(row[:timestamp]) > last_timestamp }
-
-      last_inx = filtered_df.size - 1
-
-      (0..last_inx).each do |x|
-        base_df.add_row filtered_df.row[last_inx-x]
+        aofh << HEADERS.zip(cols).to_h
       end
 
-      base_df
+      aofh
     end
   end
 end