cotcube-bardata 0.1.2 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7cc8f5a19330f55ec26073d90b2c7da30547b61903daa93f8b2977c05ad308f8
-  data.tar.gz: 3dac9e49fb0143d8dd91a86f5a7779dcb25a5c8fca719de5b23a46a5e742380f
+  metadata.gz: 7db21286fcb1a011326db3d27333b3417ec497db6c5b005250e0dcc11d355839
+  data.tar.gz: 1eca29b839ae84d43c601f793501237cabc5de25363ea1bd55f6c707393d98ba
 SHA512:
-  metadata.gz: aafb4160d474f4f4d0694faee22821f1e8a65a510dcc2c84999acce0e0f3b7511dc52cd2d28cb21856162fbed9c5fa2e9b8e65b8065b7c1603128f8ef1cdea75
-  data.tar.gz: 51ba4678419a6daf9f4d2a18810432f2f3c93c7a6417e8d0e7beae822e76ad9214450bfb1bf4479fc30c4032b51e4504c923e707f6ea8feff295f16071040df3
+  metadata.gz: 3b46cbfbd3bf8117a5076691d9eaa8985ea551d822838e69cf4336a97161e77d3c0ab5b9427f957a08a219a19290d76832527cadbf2956d16053762443e8f25b
+  data.tar.gz: b082544cd2ac1c4a36a5e23a7ce2da9764fd9ce4fa7f09a72b28a579301eb007fda3ea8c8856cb5f9a9fea6756d2c6ee2a4d84224195840793ff49d1c7985855

data/.irbrc.rb

@@ -1,5 +1,7 @@
+# frozen_string_literal: true
+
 def verbose_toggle
-  irb_context.echo ? irb_context.echo = false : irb_context.echo = true
+  irb_context.echo = (irb_context.echo ? false : true)
 end
 
 alias vt verbose_toggle

data/CHANGELOG.md

@@ -1,3 +1,41 @@
+## 0.1.7 (January 14, 2021)
+  - added :range parameter to :provide, hence to :provide_cached, :provide_daily
+  - added forgotten module_functions
+
+## 0.1.6 (January 07, 2021)
+  - prefering datetime instead date (in helpers and daily)
+  - changed keyword :set to :filter in cached, provide and trading_hours
+
+## 0.1.5 (January 02, 2021)
+  - applied some still missing cops
+
+## 0.1.4 (January 02, 2021)
+  - two minor fixes (cached.rb, daily.rb)
+  - adding first (shy) specs ... to be continued
+  - cotcube-bardata.rb: added dependency parallel, added new module files and functions
+  - provide.rb: writing provide, the central accessor to actual bardata
+  - cached.rb: implementing 'provide_cached', which manages reduced and dimished subsets of 'quarters'
+  - helpers.rb: added get_id_set
+  - daily.rb: applied cops
+  - quarters.rb: applied cops, used new get_id_set, slimmed down content in favor of 'provide'
+  - eods.rb: renamed get_id_from to get_id_set
+  - added explanation to range_matrix.rb
+  - added 'holidays' to trade_dates.csv, depending on according CSV
+  - applied cops to init.rb
+  - changed name from get_range to trading_hours
+  - minor change in gemspec
+  - fixed typos in README
+  - added bounded versions to gemspec
+  - applied cops
+  - new file trading_hours.rb providing get_range(). Based on CSV data it provides a list of ranges depicting seconds since Sunday 0:00am, which in turn can be used with the helper Array.new.to_time_interval.
+  - new file and method 'range_matrix', investigating high-low ranges of entire daily
+  - Too bad, found copied README...fixing something quite embarassing.
+
+## 0.1.3 (December 23, 2020)
+  - added .provide_most_liquids_by_eod which supports :age, filtering out files that have been updated more recently
+  - added 'type' to symbols to filter for e.g. currencies
+  - changed .provide_eods not to raise on missing eod-file but just to WARN
+
 ## 0.1.2 (December 22, 2020)
   - created and added .provide_quarters
   - added cotcube-helpers as new dependency to gempsec

data/Gemfile

@@ -1,5 +1,6 @@
-source "https://rubygems.org"
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
 
 # Specify your gem's dependencies in bitangent.gemspec
 gemspec
-

data/README.md

@@ -1,36 +1,19 @@
-# Bitangent
+# cotcube-bardata
 
-This gem provides a class (and will maybe later provide a commandline tool) to process 
-time series data into bitangents.
+This gem is a versatile provider of bardata. It relies on a directory structure most probably saved to `/var/cotcube/bardata/`. The following directories and files contain data others might expect to be delivered by a database:
 
-The underlying algorithm starts with the entire range of data and shears it (starting at 
-90 or -90 degrees) along the x axis until a bitangent is found parallel to the x axis 
-(i.e. __y == 0__ resp. __y == y.last__), resulting in an angle and a group of at least 
-2 measurements. Please note here: 
-- Measurements can be clustered to 1 finding in case there is no significant distance to 
-  the bitangent in regard to the :fuzziness, which is at least 1 'tick'.
-- Ocassionally a bitangent becomes an N-tangent, when multiple findings or clusters are 
-  in one line resp. the fuzzied ranged on the sheared graph.
-- Shearing is limited by reaching 0 degrees, so everything below the horizont (or above resp.)
-  is discarded. 
-
-After identifiying the angle of Z degress delivering N findings( actually N - 1, as the 
-last finding always is the last member of the series), the entire time series then is split  
-into N subranges, where each subrange is processed again until it reaches minimum size
-(which defaults to 3 items).
-
-Except for the very first range the challenge is to trim away the beginning of each sub
-range.
-
-The result of such a search is a tree, where it might be considerable to walk and change
-this tree by adding elements to the time series instead of recalculating it completely. 
+1. `eods`: within *eods/<id or symbol>/<date>.csv*, for each date a list of contracts is located, to be applied with the list of headers `%i[ contract date open high low close volume oi ]`.
+2. `daily`: within *daily/<id or symbol>/<contract>.csv*, for each contract all eods are provided. The list of headers is expected as `%i[ contract date open high low close volume oi ]`. Please note that it is not obvious whether `close` contains settlement or actual closing price, depending on the exchange and the broker providing the source data.
+3. `quarters`: within *quarters/<id or symbol>/<contract>.csv*, for each contract a list of quarters (15 minute intervals) is provided, depending on the first occurrence of the contract within the topN volume segment. Note the different headers here: `%i[ contract date_alike day open high low close volume ]`.
+4. `trading_hours`: within *trading_hours/<symbol or type>_<set>.csv* a list of intervals is provided, with the headers `%i[ interval_start interval_end ]` for each interval described by seconds since Sunday 0:00p.m. (as defaulted by Ruby's *DateTime.new.wday)*.
+5. `trade_dates.csv`: A growing list of trade\_dates as provided by the CME.  
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'bitangent'
+gem 'cotcube-bardata'
 ```
 
 And then execute:
@@ -39,11 +22,60 @@ And then execute:
 
 Or install it yourself as:
 
-    $ gem install bitangent
+    $ gem install cotcube-bardata
 
 ## Usage
 
-TODO: Write usage instructions here
+### Configuration
+
+The gem expects a configfile 'bardata.yml', located in '/etc/cotcube/' on linux platform and '/usr/local/etc/cotcube/' on FreeBSD. The location of the configfile can be overwritten by passing the according parameter to `init`.
+
+### daily.rb
+
+Provides
+
+* `provide_daily(symbol: nil, id: nil, contract:, timezone: Time.find_zone('America/Chicago'), config: init)`
+* `continuous(symbol: nil, id: nil, config: init, date: nil)`: Loads all dailies for given *id* and groups by date, hence providing a list of eods. 
+* `continuous_ml(symbol: nil, id: nil, base: nil)`: Provides a list of contracts, containing a list of most liquid by volume contracts as `{ date: , ml: }`
+* `continuous_actual_ml(symbol: nil, id: nil)`: Same as above, but providing the succeeding trading day (as the signaled 'ML' is yet one day old before it can be used). 
+* `continuous_overview(symbol: nil, id: nil, config: init, selector: :volume, human: false, filter: nil)`: Several purposes, but most noticeable providing the range of first and last occurrence within the top N% by volume within eods.
+
+### eods.rb
+
+Provides
+
+* `most_liquid_for(symbol: nil, id: nil, date: last_trade_date, config: init, quiet: false)`
+* `provide_most_liquids_by_eod(config: init, date: last_trade_date, filter: :volume_part, age: 1.hour)`
+* `provide_eods(**args)`
+* `provide_quarters(**args)`
+
+### quarters.rb
+
+Provides `provide_quarters(**args)`.
+
+### provide.rb
+
+Provides `provide(**args)`.
+
+### range\_matrix.rb
+
+Provides `range_matrix`, a simple method processing data based on `Bardata.continuous_actual_ml` to return a statistical overview of daily high-low ranges (not True Ranges). It contains sets for 
+
+* all available data,
+* a data subset containing the recent 12 months
+* data diminished by `:dim` (top and bottom) based on all available data
+
+and contains  *max*, *avg*, *lower*, *median*, *upper* and *max* (where 'upper' and 'lower' are like the median but at the 25percentile and 75percentile resp.).
+
+As a third dimension (sorry!) all of the above is applied to days, weeks as well as months. 
+
+### trade\_dates.rb
+
+Provides `last_trade_date`, simple fetches the current 5 trade dates from CME and returns the very last.
+
+### trading\_hours.rb
+
+Provides `get_range(symbol: nil, set: :full, force_set: false, config: init, debug: false)`, loading a set of intervals. The sets are defaulting to :full when the requested set is not found--unless :force\_set is enabled. Furthermore, if symbol is not found, the type-based version is returned. Eventually, if neither could be returned, the 24x7 interval is returned.
 
 ## Development
 
@@ -53,9 +85,10 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/bitangent.
+Bug reports and pull requests are welcome on GitHub at https://github.com/donkeybridge/bitangent.
 
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -1,6 +1,8 @@
-require "bundler/gem_tasks"
-require "rspec/core/rake_task"
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+task default: :spec

data/VERSION

@@ -1 +1 @@
-0.1.2
+0.1.7

data/cotcube-bardata.gemspec

@@ -9,13 +9,13 @@ Gem::Specification.new do |spec|
   spec.summary       = 'Functions to provide bardata; and some simple time series aggregations'
   spec.description   = 'Functions to provide bardata; and some simple time series aggregations '
 
-  spec.homepage      = 'https://github.com/donkeybridge/'+ spec.name
+  spec.homepage      = "https://github.com/donkeybridge/#{spec.name}"
   spec.license       = 'BSD-4-Clause'
   spec.required_ruby_version = Gem::Requirement.new('~> 2.7')
 
   spec.metadata['homepage_uri'] = spec.homepage
   spec.metadata['source_code_uri'] = spec.homepage
-  spec.metadata['changelog_uri'] = spec.homepage + '/CHANGELOG.md'
+  spec.metadata['changelog_uri'] = "#{spec.homepage}/CHANGELOG.md"
 
   # Specify which files should be added to the gem when it is released.
   # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
@@ -26,16 +26,15 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ['lib']
 
-  spec.add_dependency 'cotcube-indicators'
-  spec.add_dependency 'cotcube-helpers'
-  spec.add_dependency 'yaml'
-  spec.add_dependency 'activesupport'
-  spec.add_dependency 'colorize'
-  spec.add_dependency 'httparty'
-  spec.add_dependency 'rubyzip'
+  spec.add_dependency 'activesupport',      '~> 6'
+  spec.add_dependency 'colorize',           '~> 0.8'
+  spec.add_dependency 'cotcube-helpers',    '~> 0.1'
+  spec.add_dependency 'cotcube-indicators', '~> 0.1'
+  spec.add_dependency 'httparty',           '~> 0.18'
+  spec.add_dependency 'parallel',           '~> 1'
+  spec.add_dependency 'yaml',               '~> 0.1'
 
-
-  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'rake',  '>= 12'
   spec.add_development_dependency 'rspec', '~>3.6'
   spec.add_development_dependency 'yard', '~>0.9'
 end

data/lib/cotcube-bardata.rb

@@ -1,3 +1,5 @@
+# rubocop:disable Naming/FileName
+# rubocop:enable Naming/FileName
 # frozen_string_literal: true
 
 require 'active_support'
@@ -9,34 +11,62 @@ require 'date' unless defined?(DateTime)
 require 'csv'  unless defined?(CSV)
 require 'yaml' unless defined?(YAML)
 require 'cotcube-helpers'
-
-
+require 'parallel'
 
 require_relative 'cotcube-bardata/constants'
+require_relative 'cotcube-bardata/helpers'
 require_relative 'cotcube-bardata/init'
 require_relative 'cotcube-bardata/trade_dates'
 require_relative 'cotcube-bardata/daily'
 require_relative 'cotcube-bardata/quarters'
 require_relative 'cotcube-bardata/eods'
+require_relative 'cotcube-bardata/cached'
 require_relative 'cotcube-bardata/provide'
+require_relative 'cotcube-bardata/range_matrix'
+require_relative 'cotcube-bardata/trading_hours'
 
 module Cotcube
   module Bardata
-
     module_function :config_path, # provides the path of configuration directory
-      :config_prefix,             # provides the prefix of the configuration directory according to OS-specific FSH
-      :init,                      # checks whether environment is prepared and returns the config hash
-      :last_trade_date,           # Provides the most recent trade date (today or maybe last friday before weekend)
-      :provide,                   # 
-      :most_liquid_for,           # the most_liquid contract for a given symbol or id, based on date or last_trade_date
-      :provide_eods,              # provides the list of eods, either with data or just the contracts, filtered for liquidity threshold
-      :provide_daily,             # provides the list of dailies for a given symbol, which include OI. Note that the close is settlement price.
-      :continuous,                # for a given date or range, provide all contracts that exceed a given threshold of volume share
-      :continuous_overview,       # based on continuous, create list of when which contract was most liquid
-      :provide_quarters,          # provide the list of quarters, possibly as hours or days.
-      :symbols                    # reads and provides the symbols file
-    
-    # please not that module_functions of source provided in private files must be published there
+                    # provides the prefix of the configuration directory according to OS-specific FSH
+                    :config_prefix,
+                    # checks whether environment is prepared and returns the config hash
+                    :init,
+                    # Provides the most recent trade date (today or maybe last friday before weekend)
+                    :last_trade_date,
+                    :provide,
+                    # the most_liquid contract for a given symbol or id, based on date or last_trade_date
+                    :most_liquid_for,
+                    # provides the list of eods, either with data or just the contracts,
+                    # filtered for liquidity threshold
+                    :provide_eods,
+                    :provide_most_liquids_by_eod,
+                    # provides the list of dailies for a given symbol, which include OI.
+                    # Note that the close is most probably settlement price.
+                    :provide_daily,
+                    # for a given date or range, provide all contracts that exceed a given threshold of volume share
+                    :continuous,
+                    # the list of most liquid contracts (by each days volume share)
+                    :continuous_ml,
+                    # same list but riped one day each
+                    :continuous_actual_ml,
+                    # based on continuous, create list of when which contract was most liquid
+                    :continuous_overview,
+                    # provide the list of quarters, possibly as hours or days.
+                    :provide_quarters,
+                    # some statistics to estimate daily volatility of specific contract
+                    :range_matrix,
+                    # create an array of ranges based on specified source data
+                    :trading_hours,
+                    # receive id / symbol information on an uncertain set of parameters
+                    :get_id_set,
+                    :select_specific_date,
+                    :extended_select_for_range,
+                    :provide_cached,
+                    :compare,
+                    :holidays,
+                    :symbols # reads and provides the symbols file
+
+    # please note that module_functions of source provided in private files must be published there
   end
 end
-

data/lib/cotcube-bardata/cached.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+module Cotcube
+  # missing top level documentation
+  module Bardata
+    # send pre-created days based on quarters
+    def provide_cached(contract:, # rubocop:disable Metrics/ParameterLists
+                       symbol: nil, id: nil,
+                       range: nil,
+                       config: init,
+                       timezone: Time.find_zone('America/Chicago'),
+                       filter: :full, # most probably either :full or :rth
+                       force_update: false, # force reloading via provide_quarters
+                       force_recent: false) #
+
+      unless range.nil? ||
+             range.is_a?(Range) &&
+             [Date, DateTime, ActiveSupport::TimeWithZone].map do |cl|
+               (range.begin.nil? || range.begin.is_a?(cl)) &&
+               (range.end.nil? || range.end.is_a?(cl))
+             end.reduce(:|)
+        raise ArgumentError, 'Range, if given, must be either (Integer..Integer) or (Timelike..Timelike)'
+      end
+
+      unless range.nil?
+        range_begin = range.begin.nil? ? nil : timezone.parse(range.begin.to_s)
+        range_end   = range.end.nil? ? nil : timezone.parse(range.end.to_s)
+        range = (range_begin..range_end)
+      end
+
+      headers = %i[contract datetime open high low close volume]
+      sym      = get_id_set(symbol: symbol, id: id, contract: contract)
+      contract = contract[-3..]
+      dir      = "#{config[:data_path]}/cached/#{sym[:id]}_#{filter.to_s.downcase}"
+      symlink  = "#{config[:data_path]}/cached/#{sym[:symbol]}_#{filter.to_s.downcase}"
+      `mkdir -p #{dir}` unless Dir.exist? dir
+      `ln -s #{dir} #{symlink}` unless File.exist? symlink
+      file = "#{dir}/#{contract}.csv"
+      quarters_file = "#{config[:data_path]}/quarters/#{sym[:id]}/#{contract[-3..]}.csv"
+      if File.exist?(file) && (not force_update)
+        base = CSV.read(file, headers: headers).map do |x|
+          x = x.to_h
+          x[:datetime] = timezone.parse(x[:datetime])
+          %i[open high low close].each { |z| x[z] = x[z].to_f.round(9) }
+          x[:volume] = x[:volume].to_i
+          x[:type]   = "#{filter.to_s.downcase}_day".to_sym
+          x
+        end
+        if base.last[:high].zero?
+          # contract exists but is closed (has the CLOSED marker)
+          base.pop
+          # rubocop:disable Metrics/BlockNesting
+          result = if range.nil?
+                     base
+                   else
+                     base.select do |x|
+                       (range.begin.nil? ? true : x[:datetime] >= range.begin) and
+                         (range.end.nil? ? true : x[:datetime] <= range.end)
+                     end
+                   end
+          return result
+        elsif File.mtime(file) < File.mtime(quarters_file)
+          result = if range.nil?
+                     base
+                   else
+                     base.select do |x|
+                       (range.begin.nil? ? true : x[:datetime] >= range.begin) and
+                         (range.end.nil? ? true : x[:datetime] <= range.end)
+                     end
+                   end
+          # rubocop:enable Metrics/BlockNesting
+          return result
+        else
+          puts "File #{file} exists, but is neither closed nor current. Running update.".colorize(:light_green)
+        end
+      end
+      begin
+        data = provide_quarters(contract: contract, id: sym[:id], keep_marker: true)
+      rescue StandardError
+        puts "Cannot provide quarters for requested contract #{sym[:symbol]}:#{contract},"\
+            "returning '[ ]'".colorize(:light_red)
+        return []
+      end
+
+      # removing marker if existing
+      contract_is_marked = data.last[:high].zero?
+      data.pop if contract_is_marked
+      unless (filter == :full) || (data.size < 3)
+        requested_set = trading_hours(symbol: sym[:symbol], filter: filter)
+        data = data.select_within(ranges: requested_set, attr: :datetime) { |x| x.to_datetime.to_sssm }
+      end
+
+      base = Cotcube::Helpers.reduce(bars: data, to: :days)
+
+      # remove last day of result unless marked
+      base.pop unless contract_is_marked || force_recent
+
+      base.map do |x|
+        x[:datetime] = x[:datetime].to_date
+        x[:type] = "#{filter}_day".to_sym
+        x.delete(:day)
+      end
+      CSV.open(file, 'w') do |csv|
+        base.each { |b| csv << b.values_at(*headers) }
+        if contract_is_marked
+          marker = ["#{sym[:symbol]}#{contract}", base.last[:datetime] + 1.day, 0, 0, 0, 0, 0]
+          csv << marker
+        end
+      end
+      if range.nil?
+        base
+      else
+        base.select do |x|
+          (range.begin.nil? ? true : x[:date] >= range.begin) and
+            (range.end.nil? ? true : x[:date] <= range.end)
+        end
+      end
+    end
+  end
+end