cotcube-bardata 0.1.1 → 0.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c4c50df76637c9cb761c86db2b64afe372f2dd6be052a11615b9b9e961c6c890
-  data.tar.gz: 53eb198be134abc15d92e27eb34fa5b9482b964998e7016aedc9378a3199968c
+  metadata.gz: 241ac82617b49386e5ab3b2291a0d5a0e6f6f792a1184ad5956ec5eedce56d5e
+  data.tar.gz: a70bb34fe89fdc62138e2efbb8dcc7119f5e07d58672c4c9bb6923da95ab1bd1
 SHA512:
-  metadata.gz: a15393072fd65a449cdeed625409bb175c3db7c10821d4727fa2aa62577897cf7350b97520f35c70e3485bcf74eab2b8f56ae814803ff77f0276de29957e1181
-  data.tar.gz: 7c8b44f3a15abed527ff6be6c461fa786bfa2b73ca51b7aeeb718a28be38524a37788b5fab8e8513a4ede8318c134dafeda76a0171957244f75cd69668e20275
+  metadata.gz: 5e80e2f64b55ea2df59ca62d2407ea7d534568d0bcdbd75eed6c0df233fc16f9e9c0651213f0d53401a9ac336eb4fb4e4325d874fb299ce8ba01dee53c7d25d0
+  data.tar.gz: 875c0bcca1eab089286a7ecb94979db433d15d6cb5c4a45fccc46695615da632aa48f68ae3119a908287f127cee0dd93f24319e02d45103148eb7acdad992de8

data/.irbrc.rb

@@ -1,5 +1,7 @@
+# frozen_string_literal: true
+
 def verbose_toggle
-  irb_context.echo ? irb_context.echo = false : irb_context.echo = true
+  irb_context.echo = (irb_context.echo ? false : true)
 end
 
 alias vt verbose_toggle

data/CHANGELOG.md

@@ -1,3 +1,46 @@
+## 0.1.6 (January 07, 2021)
+  - prefering datetime instead date (in helpers and daily)
+  - changed keyword :set to :filter in cached, provide and trading_hours
+
+## 0.1.5 (January 02, 2021)
+  - applied some still missing cops
+
+## 0.1.4 (January 02, 2021)
+  - two minor fixes (cached.rb, daily.rb)
+  - adding first (shy) specs ... to be continued
+  - cotcube-bardata.rb: added dependency parallel, added new module files and functions
+  - provide.rb: writing provide, the central accessor to actual bardata
+  - cached.rb: implementing 'provide_cached', which manages reduced and dimished subsets of 'quarters'
+  - helpers.rb: added get_id_set
+  - daily.rb: applied cops
+  - quarters.rb: applied cops, used new get_id_set, slimmed down content in favor of 'provide'
+  - eods.rb: renamed get_id_from to get_id_set
+  - added explanation to range_matrix.rb
+  - added 'holidays' to trade_dates.csv, depending on according CSV
+  - applied cops to init.rb
+  - changed name from get_range to trading_hours
+  - minor change in gemspec
+  - fixed typos in README
+  - added bounded versions to gemspec
+  - applied cops
+  - new file trading_hours.rb providing get_range(). Based on CSV data it provides a list of ranges depicting seconds since Sunday 0:00am, which in turn can be used with the helper Array.new.to_time_interval.
+  - new file and method 'range_matrix', investigating high-low ranges of entire daily
+  - Too bad, found copied README...fixing something quite embarassing.
+
+## 0.1.3 (December 23, 2020)
+  - added .provide_most_liquids_by_eod which supports :age, filtering out files that have been updated more recently
+  - added 'type' to symbols to filter for e.g. currencies
+  - changed .provide_eods not to raise on missing eod-file but just to WARN
+
+## 0.1.2 (December 22, 2020)
+  - created and added .provide_quarters
+  - added cotcube-helpers as new dependency to gempsec
+  - added timezone to dailies and :datetime to each row
+  - updated master file with new methods and sub-file
+  - added eods with 2 simple mathods .most_liquid_for and .provide_eods
+  - added a simple getter to access CME tradedates (to fetch last_trade_date)
+  - moved daily stuff into new file, prepare .provide to become a major accessor method
+
 ## 0.1.1 (December 16, 2020)
 
 

data/Gemfile

@@ -1,5 +1,6 @@
-source "https://rubygems.org"
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
 
 # Specify your gem's dependencies in bitangent.gemspec
 gemspec
-

data/README.md

@@ -1,36 +1,19 @@
-# Bitangent
+# cotcube-bardata
 
-This gem provides a class (and will maybe later provide a commandline tool) to process 
-time series data into bitangents.
+This gem is a versatile provider of bardata. It relies on a directory structure most probably saved to `/var/cotcube/bardata/`. The following directories and files contain data others might expect to be delivered by a database:
 
-The underlying algorithm starts with the entire range of data and shears it (starting at 
-90 or -90 degrees) along the x axis until a bitangent is found parallel to the x axis 
-(i.e. __y == 0__ resp. __y == y.last__), resulting in an angle and a group of at least 
-2 measurements. Please note here: 
-- Measurements can be clustered to 1 finding in case there is no significant distance to 
-  the bitangent in regard to the :fuzziness, which is at least 1 'tick'.
-- Ocassionally a bitangent becomes an N-tangent, when multiple findings or clusters are 
-  in one line resp. the fuzzied ranged on the sheared graph.
-- Shearing is limited by reaching 0 degrees, so everything below the horizont (or above resp.)
-  is discarded. 
-
-After identifiying the angle of Z degress delivering N findings( actually N - 1, as the 
-last finding always is the last member of the series), the entire time series then is split  
-into N subranges, where each subrange is processed again until it reaches minimum size
-(which defaults to 3 items).
-
-Except for the very first range the challenge is to trim away the beginning of each sub
-range.
-
-The result of such a search is a tree, where it might be considerable to walk and change
-this tree by adding elements to the time series instead of recalculating it completely. 
+1. `eods`: within *eods/<id or symbol>/<date>.csv*, for each date a list of contracts is located, to be applied with the list of headers `%i[ contract date open high low close volume oi ]`.
+2. `daily`: within *daily/<id or symbol>/<contract>.csv*, for each contract all eods are provided. The list of headers is expected as `%i[ contract date open high low close volume oi ]`. Please note that it is not obvious whether `close` contains settlement or actual closing price, depending on the exchange and the broker providing the source data.
+3. `quarters`: within *quarters/<id or symbol>/<contract>.csv*, for each contract a list of quarters (15 minute intervals) is provided, depending on the first occurrence of the contract within the topN volume segment. Note the different headers here: `%i[ contract date_alike day open high low close volume ]`.
+4. `trading_hours`: within *trading_hours/<symbol or type>_<set>.csv* a list of intervals is provided, with the headers `%i[ interval_start interval_end ]` for each interval described by seconds since Sunday 0:00p.m. (as defaulted by Ruby's *DateTime.new.wday)*.
+5. `trade_dates.csv`: A growing list of trade\_dates as provided by the CME.  
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'bitangent'
+gem 'cotcube-bardata'
 ```
 
 And then execute:
@@ -39,11 +22,60 @@ And then execute:
 
 Or install it yourself as:
 
-    $ gem install bitangent
+    $ gem install cotcube-bardata
 
 ## Usage
 
-TODO: Write usage instructions here
+### Configuration
+
+The gem expects a configfile 'bardata.yml', located in '/etc/cotcube/' on linux platform and '/usr/local/etc/cotcube/' on FreeBSD. The location of the configfile can be overwritten by passing the according parameter to `init`.
+
+### daily.rb
+
+Provides
+
+* `provide_daily(symbol: nil, id: nil, contract:, timezone: Time.find_zone('America/Chicago'), config: init)`
+* `continuous(symbol: nil, id: nil, config: init, date: nil)`: Loads all dailies for given *id* and groups by date, hence providing a list of eods. 
+* `continuous_ml(symbol: nil, id: nil, base: nil)`: Provides a list of contracts, containing a list of most liquid by volume contracts as `{ date: , ml: }`
+* `continuous_actual_ml(symbol: nil, id: nil)`: Same as above, but providing the succeeding trading day (as the signaled 'ML' is yet one day old before it can be used). 
+* `continuous_overview(symbol: nil, id: nil, config: init, selector: :volume, human: false, filter: nil)`: Several purposes, but most noticeable providing the range of first and last occurrence within the top N% by volume within eods.
+
+### eods.rb
+
+Provides
+
+* `most_liquid_for(symbol: nil, id: nil, date: last_trade_date, config: init, quiet: false)`
+* `provide_most_liquids_by_eod(config: init, date: last_trade_date, filter: :volume_part, age: 1.hour)`
+* `provide_eods(**args)`
+* `provide_quarters(**args)`
+
+### quarters.rb
+
+Provides `provide_quarters(**args)`.
+
+### provide.rb
+
+Provides `provide(**args)`.
+
+### range\_matrix.rb
+
+Provides `range_matrix`, a simple method processing data based on `Bardata.continuous_actual_ml` to return a statistical overview of daily high-low ranges (not True Ranges). It contains sets for 
+
+* all available data,
+* a data subset containing the recent 12 months
+* data diminished by `:dim` (top and bottom) based on all available data
+
+and contains  *max*, *avg*, *lower*, *median*, *upper* and *max* (where 'upper' and 'lower' are like the median but at the 25percentile and 75percentile resp.).
+
+As a third dimension (sorry!) all of the above is applied to days, weeks as well as months. 
+
+### trade\_dates.rb
+
+Provides `last_trade_date`, simple fetches the current 5 trade dates from CME and returns the very last.
+
+### trading\_hours.rb
+
+Provides `get_range(symbol: nil, set: :full, force_set: false, config: init, debug: false)`, loading a set of intervals. The sets are defaulting to :full when the requested set is not found--unless :force\_set is enabled. Furthermore, if symbol is not found, the type-based version is returned. Eventually, if neither could be returned, the 24x7 interval is returned.
 
 ## Development
 
@@ -53,9 +85,10 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/bitangent.
+Bug reports and pull requests are welcome on GitHub at https://github.com/donkeybridge/bitangent.
 
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -1,6 +1,8 @@
-require "bundler/gem_tasks"
-require "rspec/core/rake_task"
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+task default: :spec

data/VERSION

@@ -1 +1 @@
-0.1.1
+0.1.6

data/cotcube-bardata.gemspec

@@ -9,13 +9,13 @@ Gem::Specification.new do |spec|
   spec.summary       = 'Functions to provide bardata; and some simple time series aggregations'
   spec.description   = 'Functions to provide bardata; and some simple time series aggregations '
 
-  spec.homepage      = 'https://github.com/donkeybridge/'+ spec.name
+  spec.homepage      = "https://github.com/donkeybridge/#{spec.name}"
   spec.license       = 'BSD-4-Clause'
   spec.required_ruby_version = Gem::Requirement.new('~> 2.7')
 
   spec.metadata['homepage_uri'] = spec.homepage
   spec.metadata['source_code_uri'] = spec.homepage
-  spec.metadata['changelog_uri'] = spec.homepage + '/CHANGELOG.md'
+  spec.metadata['changelog_uri'] = "#{spec.homepage}/CHANGELOG.md"
 
   # Specify which files should be added to the gem when it is released.
   # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
@@ -26,15 +26,15 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ['lib']
 
-  spec.add_dependency 'cotcube-indicators'
-  spec.add_dependency 'yaml'
-  spec.add_dependency 'activesupport'
-  spec.add_dependency 'colorize'
-  spec.add_dependency 'httparty'
-  spec.add_dependency 'rubyzip'
+  spec.add_dependency 'activesupport',      '~> 6'
+  spec.add_dependency 'colorize',           '~> 0.8'
+  spec.add_dependency 'cotcube-helpers',    '~> 0.1'
+  spec.add_dependency 'cotcube-indicators', '~> 0.1'
+  spec.add_dependency 'httparty',           '~> 0.18'
+  spec.add_dependency 'parallel',           '~> 1'
+  spec.add_dependency 'yaml',               '~> 0.1'
 
-
-  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'rake',  '>= 12'
   spec.add_development_dependency 'rspec', '~>3.6'
   spec.add_development_dependency 'yard', '~>0.9'
 end

data/lib/cotcube-bardata.rb

@@ -1,37 +1,70 @@
+# rubocop:disable Naming/FileName
+# rubocop:enable Naming/FileName
 # frozen_string_literal: true
 
 require 'active_support'
+require 'active_support/core_ext/time'
+require 'active_support/core_ext/numeric'
 require 'colorize'
+require 'httparty'
 require 'date' unless defined?(DateTime)
 require 'csv'  unless defined?(CSV)
 require 'yaml' unless defined?(YAML)
-require 'httparty'
-require 'zip'
-
+require 'cotcube-helpers'
+require 'parallel'
 
 require_relative 'cotcube-bardata/constants'
+require_relative 'cotcube-bardata/helpers'
 require_relative 'cotcube-bardata/init'
+require_relative 'cotcube-bardata/trade_dates'
+require_relative 'cotcube-bardata/daily'
+require_relative 'cotcube-bardata/quarters'
+require_relative 'cotcube-bardata/eods'
+require_relative 'cotcube-bardata/cached'
 require_relative 'cotcube-bardata/provide'
-
-private_files = Dir[__dir__ + '/cotcube-bardata/private/*.rb']
-private_files.each do |file| 
-  # puts 'Loading private module extension ' + file.chomp
-  require file.chomp
-end
+require_relative 'cotcube-bardata/range_matrix'
+require_relative 'cotcube-bardata/trading_hours'
 
 module Cotcube
   module Bardata
-
     module_function :config_path, # provides the path of configuration directory
-      :config_prefix,             # provides the prefix of the configuration directory according to OS-specific FSH
-      :init,                      # checks whether environment is prepared and returns the config hash
-      :provide,
-      :continuous,
-      :continuous_overview,
+                    # provides the prefix of the configuration directory according to OS-specific FSH
+                    :config_prefix,
+                    # checks whether environment is prepared and returns the config hash
+                    :init,
+                    # Provides the most recent trade date (today or maybe last friday before weekend)
+                    :last_trade_date,
+                    :provide,
+                    # the most_liquid contract for a given symbol or id, based on date or last_trade_date
+                    :most_liquid_for,
+                    # provides the list of eods, either with data or just the contracts,
+                    # filtered for liquidity threshold
+                    :provide_eods,
+                    :provide_most_liquids_by_eod,
+                    # provides the list of dailies for a given symbol, which include OI.
+                    # Note that the close is most probably settlement price.
+                    :provide_daily,
+                    # for a given date or range, provide all contracts that exceed a given threshold of volume share
+                    :continuous,
+                    # the list of most liquid contracts (by each days volume share)
+                    :continuous_ml,
+                    # same list but riped one day each
+                    :continuous_actual_ml,
+                    # based on continuous, create list of when which contract was most liquid
+                    :continuous_overview,
+                    # provide the list of quarters, possibly as hours or days.
+                    :provide_quarters,
+                    # some statistics to estimate daily volatility of specific contract
+                    :range_matrix,
+                    # create an array of ranges based on specified source data
+                    :trading_hours,
+                    # receive id / symbol information on an uncertain set of parameters
+                    :get_id_set,
+                    :provide_cached,
+                    :compare,
+                    :holidays,
+                    :symbols # reads and provides the symbols file
 
-      :symbols                    # reads and provides the symbols file
-    
-    # please not that module_functions of source provided in private files must be published there
+    # please note that module_functions of source provided in private files must be published there
   end
 end
-

data/lib/cotcube-bardata/cached.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+module Cotcube
+  # missing top level documentation
+  module Bardata
+    # send pre-created days based on quarters
+    def provide_cached(contract:, # rubocop:disable Metrics/ParameterLists
+                       symbol: nil, id: nil,
+                       config: init,
+                       timezone: Time.find_zone('America/Chicago'),
+                       filter: :full, # most probably either :full or :rth
+                       force_update: false, # force reloading via provide_quarters
+                       force_recent: false) #
+
+      headers = %i[contract datetime open high low close volume]
+      sym      = get_id_set(symbol: symbol, id: id, contract: contract)
+      contract = contract[-3..]
+      dir      = "#{config[:data_path]}/cached/#{sym[:id]}_#{filter.to_s.downcase}"
+      symlink  = "#{config[:data_path]}/cached/#{sym[:symbol]}_#{filter.to_s.downcase}"
+      `mkdir -p #{dir}` unless Dir.exist? dir
+      `ln -s #{dir} #{symlink}` unless File.exist? symlink
+      file = "#{dir}/#{contract}.csv"
+      quarters_file = "#{config[:data_path]}/quarters/#{sym[:id]}/#{contract[-3..]}.csv"
+      if File.exist?(file) && (not force_update)
+        base = CSV.read(file, headers: headers).map do |x|
+          x = x.to_h
+          x[:datetime] = timezone.parse(x[:datetime])
+          %i[open high low close].each { |z| x[z] = x[z].to_f.round(9) }
+          x[:volume] = x[:volume].to_i
+          x[:type]   = "#{filter.to_s.downcase}_day".to_sym
+          x
+        end
+        if base.last[:high].zero?
+          # contract exists but is closed (has the CLOSED marker)
+          base.pop
+          return base
+        elsif File.mtime(file) < File.mtime(quarters_file)
+          return base
+        else
+          puts "File #{file} exists, but is neither closed nor current. Running update.".colorize(:light_green)
+        end
+      end
+      begin
+        data = provide_quarters(contract: contract, id: sym[:id], keep_marker: true)
+      rescue StandardError
+        puts "Cannot provide quarters for requested contract #{sym[:symbol]}:#{contract},"\
+             "returning '[ ]'".colorize(:light_red)
+        return []
+      end
+
+      # removing marker if existing
+      contract_is_marked = data.last[:high].zero?
+      data.pop if contract_is_marked
+      unless (filter == :full) || (data.size < 3)
+        requested_set = trading_hours(symbol: sym[:symbol], filter: filter)
+        data = data.select_within(ranges: requested_set, attr: :datetime) { |x| x.to_datetime.to_sssm }
+      end
+
+      base = Cotcube::Helpers.reduce(bars: data, to: :days)
+
+      # remove last day of result unless marked
+      base.pop unless contract_is_marked || force_recent
+
+      base.map do |x|
+        x[:datetime] = x[:datetime].to_date
+        x[:type] = "#{filter}_day".to_sym
+        x.delete(:day)
+      end
+      CSV.open(file, 'w') do |csv|
+        base.each { |b| csv << b.values_at(*headers) }
+        if contract_is_marked
+          marker = ["#{sym[:symbol]}#{contract}", base.last[:datetime] + 1.day, 0, 0, 0, 0, 0]
+          csv << marker
+        end
+      end
+      base
+    end
+  end
+end

data/lib/cotcube-bardata/constants.rb

@@ -2,16 +2,16 @@
 
 module Cotcube
   module Bardata
-
     SYMBOL_EXAMPLES = [
-      { id: "13874U", symbol: "ET", ticksize: 0.25, power: 1.25, months: "HMUZ", bcf: 1.0, reports: "LF", name: "S&P 500 MICRO" },
-      { id: "209747", symbol: "NM", ticksize: 0.25, power: 0.5,  monhts: "HMUZ", bcf: 1.0, reports: "LF", name: "NASDAQ 100 MICRO" }
+      { id: '13874U', symbol: 'ET', ticksize: 0.25, power: 1.25, months: 'HMUZ', bcf: 1.0, reports: 'LF',
+        name: 'S&P 500 MICRO' },
+      { id: '209747', symbol: 'NM', ticksize: 0.25, power: 0.5,  months: 'HMUZ', bcf: 1.0, reports: 'LF',
+        name: 'NASDAQ 100 MICRO' }
     ].freeze
 
-    MONTH_COLOURS  = { 'F' => :cyan,  'G' => :green,   'H' => :light_green,
-                       'J' => :blue,  'K' => :yellow,  'M' => :light_yellow,
-                       'N' => :cyan,  'Q' => :magenta, 'U' => :light_magenta,
-                       'V' => :blue,  'X' => :red,     'Z' => :light_red }.freeze
-
+    MONTH_COLOURS = { 'F' => :cyan, 'G' => :green, 'H' => :light_green,
+                      'J' => :blue,  'K' => :yellow,  'M' => :light_yellow,
+                      'N' => :cyan,  'Q' => :magenta, 'U' => :light_magenta,
+                      'V' => :blue,  'X' => :red,     'Z' => :light_red }.freeze
   end
 end