cotcube-bardata 0.1.1 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c4c50df76637c9cb761c86db2b64afe372f2dd6be052a11615b9b9e961c6c890
4
- data.tar.gz: 53eb198be134abc15d92e27eb34fa5b9482b964998e7016aedc9378a3199968c
3
+ metadata.gz: 241ac82617b49386e5ab3b2291a0d5a0e6f6f792a1184ad5956ec5eedce56d5e
4
+ data.tar.gz: a70bb34fe89fdc62138e2efbb8dcc7119f5e07d58672c4c9bb6923da95ab1bd1
5
5
  SHA512:
6
- metadata.gz: a15393072fd65a449cdeed625409bb175c3db7c10821d4727fa2aa62577897cf7350b97520f35c70e3485bcf74eab2b8f56ae814803ff77f0276de29957e1181
7
- data.tar.gz: 7c8b44f3a15abed527ff6be6c461fa786bfa2b73ca51b7aeeb718a28be38524a37788b5fab8e8513a4ede8318c134dafeda76a0171957244f75cd69668e20275
6
+ metadata.gz: 5e80e2f64b55ea2df59ca62d2407ea7d534568d0bcdbd75eed6c0df233fc16f9e9c0651213f0d53401a9ac336eb4fb4e4325d874fb299ce8ba01dee53c7d25d0
7
+ data.tar.gz: 875c0bcca1eab089286a7ecb94979db433d15d6cb5c4a45fccc46695615da632aa48f68ae3119a908287f127cee0dd93f24319e02d45103148eb7acdad992de8
data/.irbrc.rb CHANGED
@@ -1,5 +1,7 @@
1
+ # frozen_string_literal: true
2
+
1
3
  def verbose_toggle
2
- irb_context.echo ? irb_context.echo = false : irb_context.echo = true
4
+ irb_context.echo = (irb_context.echo ? false : true)
3
5
  end
4
6
 
5
7
  alias vt verbose_toggle
@@ -1,3 +1,46 @@
1
+ ## 0.1.6 (January 07, 2021)
2
+ - prefering datetime instead date (in helpers and daily)
3
+ - changed keyword :set to :filter in cached, provide and trading_hours
4
+
5
+ ## 0.1.5 (January 02, 2021)
6
+ - applied some still missing cops
7
+
8
+ ## 0.1.4 (January 02, 2021)
9
+ - two minor fixes (cached.rb, daily.rb)
10
+ - adding first (shy) specs ... to be continued
11
+ - cotcube-bardata.rb: added dependency parallel, added new module files and functions
12
+ - provide.rb: writing provide, the central accessor to actual bardata
13
+ - cached.rb: implementing 'provide_cached', which manages reduced and dimished subsets of 'quarters'
14
+ - helpers.rb: added get_id_set
15
+ - daily.rb: applied cops
16
+ - quarters.rb: applied cops, used new get_id_set, slimmed down content in favor of 'provide'
17
+ - eods.rb: renamed get_id_from to get_id_set
18
+ - added explanation to range_matrix.rb
19
+ - added 'holidays' to trade_dates.csv, depending on according CSV
20
+ - applied cops to init.rb
21
+ - changed name from get_range to trading_hours
22
+ - minor change in gemspec
23
+ - fixed typos in README
24
+ - added bounded versions to gemspec
25
+ - applied cops
26
+ - new file trading_hours.rb providing get_range(). Based on CSV data it provides a list of ranges depicting seconds since Sunday 0:00am, which in turn can be used with the helper Array.new.to_time_interval.
27
+ - new file and method 'range_matrix', investigating high-low ranges of entire daily
28
+ - Too bad, found copied README...fixing something quite embarassing.
29
+
30
+ ## 0.1.3 (December 23, 2020)
31
+ - added .provide_most_liquids_by_eod which supports :age, filtering out files that have been updated more recently
32
+ - added 'type' to symbols to filter for e.g. currencies
33
+ - changed .provide_eods not to raise on missing eod-file but just to WARN
34
+
35
+ ## 0.1.2 (December 22, 2020)
36
+ - created and added .provide_quarters
37
+ - added cotcube-helpers as new dependency to gempsec
38
+ - added timezone to dailies and :datetime to each row
39
+ - updated master file with new methods and sub-file
40
+ - added eods with 2 simple mathods .most_liquid_for and .provide_eods
41
+ - added a simple getter to access CME tradedates (to fetch last_trade_date)
42
+ - moved daily stuff into new file, prepare .provide to become a major accessor method
43
+
1
44
  ## 0.1.1 (December 16, 2020)
2
45
 
3
46
 
data/Gemfile CHANGED
@@ -1,5 +1,6 @@
1
- source "https://rubygems.org"
1
+ # frozen_string_literal: true
2
+
3
+ source 'https://rubygems.org'
2
4
 
3
5
  # Specify your gem's dependencies in bitangent.gemspec
4
6
  gemspec
5
-
data/README.md CHANGED
@@ -1,36 +1,19 @@
1
- # Bitangent
1
+ # cotcube-bardata
2
2
 
3
- This gem provides a class (and will maybe later provide a commandline tool) to process
4
- time series data into bitangents.
3
+ This gem is a versatile provider of bardata. It relies on a directory structure most probably saved to `/var/cotcube/bardata/`. The following directories and files contain data others might expect to be delivered by a database:
5
4
 
6
- The underlying algorithm starts with the entire range of data and shears it (starting at
7
- 90 or -90 degrees) along the x axis until a bitangent is found parallel to the x axis
8
- (i.e. __y == 0__ resp. __y == y.last__), resulting in an angle and a group of at least
9
- 2 measurements. Please note here:
10
- - Measurements can be clustered to 1 finding in case there is no significant distance to
11
- the bitangent in regard to the :fuzziness, which is at least 1 'tick'.
12
- - Ocassionally a bitangent becomes an N-tangent, when multiple findings or clusters are
13
- in one line resp. the fuzzied ranged on the sheared graph.
14
- - Shearing is limited by reaching 0 degrees, so everything below the horizont (or above resp.)
15
- is discarded.
16
-
17
- After identifiying the angle of Z degress delivering N findings( actually N - 1, as the
18
- last finding always is the last member of the series), the entire time series then is split
19
- into N subranges, where each subrange is processed again until it reaches minimum size
20
- (which defaults to 3 items).
21
-
22
- Except for the very first range the challenge is to trim away the beginning of each sub
23
- range.
24
-
25
- The result of such a search is a tree, where it might be considerable to walk and change
26
- this tree by adding elements to the time series instead of recalculating it completely.
5
+ 1. `eods`: within *eods/<id or symbol>/<date>.csv*, for each date a list of contracts is located, to be applied with the list of headers `%i[ contract date open high low close volume oi ]`.
6
+ 2. `daily`: within *daily/<id or symbol>/<contract>.csv*, for each contract all eods are provided. The list of headers is expected as `%i[ contract date open high low close volume oi ]`. Please note that it is not obvious whether `close` contains settlement or actual closing price, depending on the exchange and the broker providing the source data.
7
+ 3. `quarters`: within *quarters/<id or symbol>/<contract>.csv*, for each contract a list of quarters (15 minute intervals) is provided, depending on the first occurrence of the contract within the topN volume segment. Note the different headers here: `%i[ contract date_alike day open high low close volume ]`.
8
+ 4. `trading_hours`: within *trading_hours/<symbol or type>_<set>.csv* a list of intervals is provided, with the headers `%i[ interval_start interval_end ]` for each interval described by seconds since Sunday 0:00p.m. (as defaulted by Ruby's *DateTime.new.wday)*.
9
+ 5. `trade_dates.csv`: A growing list of trade\_dates as provided by the CME.
27
10
 
28
11
  ## Installation
29
12
 
30
13
  Add this line to your application's Gemfile:
31
14
 
32
15
  ```ruby
33
- gem 'bitangent'
16
+ gem 'cotcube-bardata'
34
17
  ```
35
18
 
36
19
  And then execute:
@@ -39,11 +22,60 @@ And then execute:
39
22
 
40
23
  Or install it yourself as:
41
24
 
42
- $ gem install bitangent
25
+ $ gem install cotcube-bardata
43
26
 
44
27
  ## Usage
45
28
 
46
- TODO: Write usage instructions here
29
+ ### Configuration
30
+
31
+ The gem expects a configfile 'bardata.yml', located in '/etc/cotcube/' on linux platform and '/usr/local/etc/cotcube/' on FreeBSD. The location of the configfile can be overwritten by passing the according parameter to `init`.
32
+
33
+ ### daily.rb
34
+
35
+ Provides
36
+
37
+ * `provide_daily(symbol: nil, id: nil, contract:, timezone: Time.find_zone('America/Chicago'), config: init)`
38
+ * `continuous(symbol: nil, id: nil, config: init, date: nil)`: Loads all dailies for given *id* and groups by date, hence providing a list of eods.
39
+ * `continuous_ml(symbol: nil, id: nil, base: nil)`: Provides a list of contracts, containing a list of most liquid by volume contracts as `{ date: , ml: }`
40
+ * `continuous_actual_ml(symbol: nil, id: nil)`: Same as above, but providing the succeeding trading day (as the signaled 'ML' is yet one day old before it can be used).
41
+ * `continuous_overview(symbol: nil, id: nil, config: init, selector: :volume, human: false, filter: nil)`: Several purposes, but most noticeable providing the range of first and last occurrence within the top N% by volume within eods.
42
+
43
+ ### eods.rb
44
+
45
+ Provides
46
+
47
+ * `most_liquid_for(symbol: nil, id: nil, date: last_trade_date, config: init, quiet: false)`
48
+ * `provide_most_liquids_by_eod(config: init, date: last_trade_date, filter: :volume_part, age: 1.hour)`
49
+ * `provide_eods(**args)`
50
+ * `provide_quarters(**args)`
51
+
52
+ ### quarters.rb
53
+
54
+ Provides `provide_quarters(**args)`.
55
+
56
+ ### provide.rb
57
+
58
+ Provides `provide(**args)`.
59
+
60
+ ### range\_matrix.rb
61
+
62
+ Provides `range_matrix`, a simple method processing data based on `Bardata.continuous_actual_ml` to return a statistical overview of daily high-low ranges (not True Ranges). It contains sets for
63
+
64
+ * all available data,
65
+ * a data subset containing the recent 12 months
66
+ * data diminished by `:dim` (top and bottom) based on all available data
67
+
68
+ and contains *max*, *avg*, *lower*, *median*, *upper* and *max* (where 'upper' and 'lower' are like the median but at the 25percentile and 75percentile resp.).
69
+
70
+ As a third dimension (sorry!) all of the above is applied to days, weeks as well as months.
71
+
72
+ ### trade\_dates.rb
73
+
74
+ Provides `last_trade_date`, simple fetches the current 5 trade dates from CME and returns the very last.
75
+
76
+ ### trading\_hours.rb
77
+
78
+ Provides `get_range(symbol: nil, set: :full, force_set: false, config: init, debug: false)`, loading a set of intervals. The sets are defaulting to :full when the requested set is not found--unless :force\_set is enabled. Furthermore, if symbol is not found, the type-based version is returned. Eventually, if neither could be returned, the 24x7 interval is returned.
47
79
 
48
80
  ## Development
49
81
 
@@ -53,9 +85,10 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
53
85
 
54
86
  ## Contributing
55
87
 
56
- Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/bitangent.
88
+ Bug reports and pull requests are welcome on GitHub at https://github.com/donkeybridge/bitangent.
57
89
 
58
90
 
59
91
  ## License
60
92
 
61
93
  The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
94
+
data/Rakefile CHANGED
@@ -1,6 +1,8 @@
1
- require "bundler/gem_tasks"
2
- require "rspec/core/rake_task"
1
+ # frozen_string_literal: true
2
+
3
+ require 'bundler/gem_tasks'
4
+ require 'rspec/core/rake_task'
3
5
 
4
6
  RSpec::Core::RakeTask.new(:spec)
5
7
 
6
- task :default => :spec
8
+ task default: :spec
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.1.1
1
+ 0.1.6
@@ -9,13 +9,13 @@ Gem::Specification.new do |spec|
9
9
  spec.summary = 'Functions to provide bardata; and some simple time series aggregations'
10
10
  spec.description = 'Functions to provide bardata; and some simple time series aggregations '
11
11
 
12
- spec.homepage = 'https://github.com/donkeybridge/'+ spec.name
12
+ spec.homepage = "https://github.com/donkeybridge/#{spec.name}"
13
13
  spec.license = 'BSD-4-Clause'
14
14
  spec.required_ruby_version = Gem::Requirement.new('~> 2.7')
15
15
 
16
16
  spec.metadata['homepage_uri'] = spec.homepage
17
17
  spec.metadata['source_code_uri'] = spec.homepage
18
- spec.metadata['changelog_uri'] = spec.homepage + '/CHANGELOG.md'
18
+ spec.metadata['changelog_uri'] = "#{spec.homepage}/CHANGELOG.md"
19
19
 
20
20
  # Specify which files should be added to the gem when it is released.
21
21
  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
@@ -26,15 +26,15 @@ Gem::Specification.new do |spec|
26
26
  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
27
27
  spec.require_paths = ['lib']
28
28
 
29
- spec.add_dependency 'cotcube-indicators'
30
- spec.add_dependency 'yaml'
31
- spec.add_dependency 'activesupport'
32
- spec.add_dependency 'colorize'
33
- spec.add_dependency 'httparty'
34
- spec.add_dependency 'rubyzip'
29
+ spec.add_dependency 'activesupport', '~> 6'
30
+ spec.add_dependency 'colorize', '~> 0.8'
31
+ spec.add_dependency 'cotcube-helpers', '~> 0.1'
32
+ spec.add_dependency 'cotcube-indicators', '~> 0.1'
33
+ spec.add_dependency 'httparty', '~> 0.18'
34
+ spec.add_dependency 'parallel', '~> 1'
35
+ spec.add_dependency 'yaml', '~> 0.1'
35
36
 
36
-
37
- spec.add_development_dependency 'rake'
37
+ spec.add_development_dependency 'rake', '>= 12'
38
38
  spec.add_development_dependency 'rspec', '~>3.6'
39
39
  spec.add_development_dependency 'yard', '~>0.9'
40
40
  end
@@ -1,37 +1,70 @@
1
+ # rubocop:disable Naming/FileName
2
+ # rubocop:enable Naming/FileName
1
3
  # frozen_string_literal: true
2
4
 
3
5
  require 'active_support'
6
+ require 'active_support/core_ext/time'
7
+ require 'active_support/core_ext/numeric'
4
8
  require 'colorize'
9
+ require 'httparty'
5
10
  require 'date' unless defined?(DateTime)
6
11
  require 'csv' unless defined?(CSV)
7
12
  require 'yaml' unless defined?(YAML)
8
- require 'httparty'
9
- require 'zip'
10
-
13
+ require 'cotcube-helpers'
14
+ require 'parallel'
11
15
 
12
16
  require_relative 'cotcube-bardata/constants'
17
+ require_relative 'cotcube-bardata/helpers'
13
18
  require_relative 'cotcube-bardata/init'
19
+ require_relative 'cotcube-bardata/trade_dates'
20
+ require_relative 'cotcube-bardata/daily'
21
+ require_relative 'cotcube-bardata/quarters'
22
+ require_relative 'cotcube-bardata/eods'
23
+ require_relative 'cotcube-bardata/cached'
14
24
  require_relative 'cotcube-bardata/provide'
15
-
16
- private_files = Dir[__dir__ + '/cotcube-bardata/private/*.rb']
17
- private_files.each do |file|
18
- # puts 'Loading private module extension ' + file.chomp
19
- require file.chomp
20
- end
25
+ require_relative 'cotcube-bardata/range_matrix'
26
+ require_relative 'cotcube-bardata/trading_hours'
21
27
 
22
28
  module Cotcube
23
29
  module Bardata
24
-
25
30
  module_function :config_path, # provides the path of configuration directory
26
- :config_prefix, # provides the prefix of the configuration directory according to OS-specific FSH
27
- :init, # checks whether environment is prepared and returns the config hash
28
- :provide,
29
- :continuous,
30
- :continuous_overview,
31
+ # provides the prefix of the configuration directory according to OS-specific FSH
32
+ :config_prefix,
33
+ # checks whether environment is prepared and returns the config hash
34
+ :init,
35
+ # Provides the most recent trade date (today or maybe last friday before weekend)
36
+ :last_trade_date,
37
+ :provide,
38
+ # the most_liquid contract for a given symbol or id, based on date or last_trade_date
39
+ :most_liquid_for,
40
+ # provides the list of eods, either with data or just the contracts,
41
+ # filtered for liquidity threshold
42
+ :provide_eods,
43
+ :provide_most_liquids_by_eod,
44
+ # provides the list of dailies for a given symbol, which include OI.
45
+ # Note that the close is most probably settlement price.
46
+ :provide_daily,
47
+ # for a given date or range, provide all contracts that exceed a given threshold of volume share
48
+ :continuous,
49
+ # the list of most liquid contracts (by each days volume share)
50
+ :continuous_ml,
51
+ # same list but riped one day each
52
+ :continuous_actual_ml,
53
+ # based on continuous, create list of when which contract was most liquid
54
+ :continuous_overview,
55
+ # provide the list of quarters, possibly as hours or days.
56
+ :provide_quarters,
57
+ # some statistics to estimate daily volatility of specific contract
58
+ :range_matrix,
59
+ # create an array of ranges based on specified source data
60
+ :trading_hours,
61
+ # receive id / symbol information on an uncertain set of parameters
62
+ :get_id_set,
63
+ :provide_cached,
64
+ :compare,
65
+ :holidays,
66
+ :symbols # reads and provides the symbols file
31
67
 
32
- :symbols # reads and provides the symbols file
33
-
34
- # please not that module_functions of source provided in private files must be published there
68
+ # please note that module_functions of source provided in private files must be published there
35
69
  end
36
70
  end
37
-
@@ -0,0 +1,79 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Cotcube
4
+ # missing top level documentation
5
+ module Bardata
6
+ # send pre-created days based on quarters
7
+ def provide_cached(contract:, # rubocop:disable Metrics/ParameterLists
8
+ symbol: nil, id: nil,
9
+ config: init,
10
+ timezone: Time.find_zone('America/Chicago'),
11
+ filter: :full, # most probably either :full or :rth
12
+ force_update: false, # force reloading via provide_quarters
13
+ force_recent: false) #
14
+
15
+ headers = %i[contract datetime open high low close volume]
16
+ sym = get_id_set(symbol: symbol, id: id, contract: contract)
17
+ contract = contract[-3..]
18
+ dir = "#{config[:data_path]}/cached/#{sym[:id]}_#{filter.to_s.downcase}"
19
+ symlink = "#{config[:data_path]}/cached/#{sym[:symbol]}_#{filter.to_s.downcase}"
20
+ `mkdir -p #{dir}` unless Dir.exist? dir
21
+ `ln -s #{dir} #{symlink}` unless File.exist? symlink
22
+ file = "#{dir}/#{contract}.csv"
23
+ quarters_file = "#{config[:data_path]}/quarters/#{sym[:id]}/#{contract[-3..]}.csv"
24
+ if File.exist?(file) && (not force_update)
25
+ base = CSV.read(file, headers: headers).map do |x|
26
+ x = x.to_h
27
+ x[:datetime] = timezone.parse(x[:datetime])
28
+ %i[open high low close].each { |z| x[z] = x[z].to_f.round(9) }
29
+ x[:volume] = x[:volume].to_i
30
+ x[:type] = "#{filter.to_s.downcase}_day".to_sym
31
+ x
32
+ end
33
+ if base.last[:high].zero?
34
+ # contract exists but is closed (has the CLOSED marker)
35
+ base.pop
36
+ return base
37
+ elsif File.mtime(file) < File.mtime(quarters_file)
38
+ return base
39
+ else
40
+ puts "File #{file} exists, but is neither closed nor current. Running update.".colorize(:light_green)
41
+ end
42
+ end
43
+ begin
44
+ data = provide_quarters(contract: contract, id: sym[:id], keep_marker: true)
45
+ rescue StandardError
46
+ puts "Cannot provide quarters for requested contract #{sym[:symbol]}:#{contract},"\
47
+ "returning '[ ]'".colorize(:light_red)
48
+ return []
49
+ end
50
+
51
+ # removing marker if existing
52
+ contract_is_marked = data.last[:high].zero?
53
+ data.pop if contract_is_marked
54
+ unless (filter == :full) || (data.size < 3)
55
+ requested_set = trading_hours(symbol: sym[:symbol], filter: filter)
56
+ data = data.select_within(ranges: requested_set, attr: :datetime) { |x| x.to_datetime.to_sssm }
57
+ end
58
+
59
+ base = Cotcube::Helpers.reduce(bars: data, to: :days)
60
+
61
+ # remove last day of result unless marked
62
+ base.pop unless contract_is_marked || force_recent
63
+
64
+ base.map do |x|
65
+ x[:datetime] = x[:datetime].to_date
66
+ x[:type] = "#{filter}_day".to_sym
67
+ x.delete(:day)
68
+ end
69
+ CSV.open(file, 'w') do |csv|
70
+ base.each { |b| csv << b.values_at(*headers) }
71
+ if contract_is_marked
72
+ marker = ["#{sym[:symbol]}#{contract}", base.last[:datetime] + 1.day, 0, 0, 0, 0, 0]
73
+ csv << marker
74
+ end
75
+ end
76
+ base
77
+ end
78
+ end
79
+ end
@@ -2,16 +2,16 @@
2
2
 
3
3
  module Cotcube
4
4
  module Bardata
5
-
6
5
  SYMBOL_EXAMPLES = [
7
- { id: "13874U", symbol: "ET", ticksize: 0.25, power: 1.25, months: "HMUZ", bcf: 1.0, reports: "LF", name: "S&P 500 MICRO" },
8
- { id: "209747", symbol: "NM", ticksize: 0.25, power: 0.5, monhts: "HMUZ", bcf: 1.0, reports: "LF", name: "NASDAQ 100 MICRO" }
6
+ { id: '13874U', symbol: 'ET', ticksize: 0.25, power: 1.25, months: 'HMUZ', bcf: 1.0, reports: 'LF',
7
+ name: 'S&P 500 MICRO' },
8
+ { id: '209747', symbol: 'NM', ticksize: 0.25, power: 0.5, months: 'HMUZ', bcf: 1.0, reports: 'LF',
9
+ name: 'NASDAQ 100 MICRO' }
9
10
  ].freeze
10
11
 
11
- MONTH_COLOURS = { 'F' => :cyan, 'G' => :green, 'H' => :light_green,
12
- 'J' => :blue, 'K' => :yellow, 'M' => :light_yellow,
13
- 'N' => :cyan, 'Q' => :magenta, 'U' => :light_magenta,
14
- 'V' => :blue, 'X' => :red, 'Z' => :light_red }.freeze
15
-
12
+ MONTH_COLOURS = { 'F' => :cyan, 'G' => :green, 'H' => :light_green,
13
+ 'J' => :blue, 'K' => :yellow, 'M' => :light_yellow,
14
+ 'N' => :cyan, 'Q' => :magenta, 'U' => :light_magenta,
15
+ 'V' => :blue, 'X' => :red, 'Z' => :light_red }.freeze
16
16
  end
17
17
  end