sqa 0.0.24 → 0.0.31

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (180) hide show
  1. checksums.yaml +4 -4
  2. data/.goose/memory/development.txt +3 -0
  3. data/.semver +6 -0
  4. data/ARCHITECTURE.md +648 -0
  5. data/CHANGELOG.md +82 -0
  6. data/CLAUDE.md +653 -0
  7. data/COMMITS.md +196 -0
  8. data/DATAFRAME_ARCHITECTURE_REVIEW.md +421 -0
  9. data/NEXT-STEPS.md +154 -0
  10. data/README.md +812 -262
  11. data/TASKS.md +358 -0
  12. data/TEST_RESULTS.md +140 -0
  13. data/TODO.md +42 -0
  14. data/_notes.txt +25 -0
  15. data/bin/sqa-console +11 -0
  16. data/data/talk_talk.json +103284 -0
  17. data/develop_summary.md +313 -0
  18. data/docs/advanced/backtesting.md +206 -0
  19. data/docs/advanced/ensemble.md +68 -0
  20. data/docs/advanced/fpop.md +153 -0
  21. data/docs/advanced/index.md +112 -0
  22. data/docs/advanced/multi-timeframe.md +67 -0
  23. data/docs/advanced/pattern-matcher.md +75 -0
  24. data/docs/advanced/portfolio-optimizer.md +79 -0
  25. data/docs/advanced/portfolio.md +166 -0
  26. data/docs/advanced/risk-management.md +210 -0
  27. data/docs/advanced/strategy-generator.md +158 -0
  28. data/docs/advanced/streaming.md +209 -0
  29. data/docs/ai_and_ml.md +80 -0
  30. data/docs/api/dataframe.md +1115 -0
  31. data/docs/api/index.md +126 -0
  32. data/docs/assets/css/custom.css +88 -0
  33. data/docs/assets/js/mathjax.js +18 -0
  34. data/docs/concepts/index.md +68 -0
  35. data/docs/contributing/index.md +60 -0
  36. data/docs/data-sources/index.md +66 -0
  37. data/docs/data_frame.md +317 -97
  38. data/docs/factors_that_impact_price.md +26 -0
  39. data/docs/finviz.md +11 -0
  40. data/docs/fx_pro_bit.md +25 -0
  41. data/docs/genetic_programming.md +104 -0
  42. data/docs/getting-started/index.md +123 -0
  43. data/docs/getting-started/installation.md +229 -0
  44. data/docs/getting-started/quick-start.md +244 -0
  45. data/docs/i_gotta_an_idea.md +22 -0
  46. data/docs/index.md +163 -0
  47. data/docs/indicators/index.md +97 -0
  48. data/docs/indicators.md +110 -24
  49. data/docs/options.md +8 -0
  50. data/docs/strategies/bollinger-bands.md +146 -0
  51. data/docs/strategies/consensus.md +64 -0
  52. data/docs/strategies/custom.md +310 -0
  53. data/docs/strategies/ema.md +53 -0
  54. data/docs/strategies/index.md +92 -0
  55. data/docs/strategies/kbs.md +164 -0
  56. data/docs/strategies/macd.md +96 -0
  57. data/docs/strategies/market-profile.md +54 -0
  58. data/docs/strategies/mean-reversion.md +58 -0
  59. data/docs/strategies/rsi.md +95 -0
  60. data/docs/strategies/sma.md +55 -0
  61. data/docs/strategies/stochastic.md +63 -0
  62. data/docs/strategies/volume-breakout.md +54 -0
  63. data/docs/tags.md +7 -0
  64. data/docs/true_strength_index.md +46 -0
  65. data/docs/weighted_moving_average.md +48 -0
  66. data/examples/README.md +354 -0
  67. data/examples/advanced_features_example.rb +350 -0
  68. data/examples/fpop_analysis_example.rb +191 -0
  69. data/examples/genetic_programming_example.rb +148 -0
  70. data/examples/kbs_strategy_example.rb +208 -0
  71. data/examples/pattern_context_example.rb +300 -0
  72. data/examples/rails_app/Gemfile +34 -0
  73. data/examples/rails_app/README.md +416 -0
  74. data/examples/rails_app/app/assets/javascripts/application.js +107 -0
  75. data/examples/rails_app/app/assets/stylesheets/application.css +659 -0
  76. data/examples/rails_app/app/controllers/analysis_controller.rb +11 -0
  77. data/examples/rails_app/app/controllers/api/v1/stocks_controller.rb +227 -0
  78. data/examples/rails_app/app/controllers/application_controller.rb +22 -0
  79. data/examples/rails_app/app/controllers/backtest_controller.rb +11 -0
  80. data/examples/rails_app/app/controllers/dashboard_controller.rb +21 -0
  81. data/examples/rails_app/app/controllers/portfolio_controller.rb +7 -0
  82. data/examples/rails_app/app/views/analysis/show.html.erb +209 -0
  83. data/examples/rails_app/app/views/backtest/show.html.erb +171 -0
  84. data/examples/rails_app/app/views/dashboard/index.html.erb +118 -0
  85. data/examples/rails_app/app/views/dashboard/show.html.erb +408 -0
  86. data/examples/rails_app/app/views/errors/show.html.erb +17 -0
  87. data/examples/rails_app/app/views/layouts/application.html.erb +60 -0
  88. data/examples/rails_app/app/views/portfolio/index.html.erb +33 -0
  89. data/examples/rails_app/bin/rails +6 -0
  90. data/examples/rails_app/config/application.rb +45 -0
  91. data/examples/rails_app/config/boot.rb +5 -0
  92. data/examples/rails_app/config/database.yml +18 -0
  93. data/examples/rails_app/config/environment.rb +11 -0
  94. data/examples/rails_app/config/routes.rb +26 -0
  95. data/examples/rails_app/config.ru +8 -0
  96. data/examples/realtime_stream_example.rb +274 -0
  97. data/examples/sinatra_app/Gemfile +22 -0
  98. data/examples/sinatra_app/QUICKSTART.md +159 -0
  99. data/examples/sinatra_app/README.md +461 -0
  100. data/examples/sinatra_app/app.rb +344 -0
  101. data/examples/sinatra_app/config.ru +5 -0
  102. data/examples/sinatra_app/public/css/style.css +659 -0
  103. data/examples/sinatra_app/public/js/app.js +107 -0
  104. data/examples/sinatra_app/views/analyze.erb +306 -0
  105. data/examples/sinatra_app/views/backtest.erb +325 -0
  106. data/examples/sinatra_app/views/dashboard.erb +419 -0
  107. data/examples/sinatra_app/views/error.erb +58 -0
  108. data/examples/sinatra_app/views/index.erb +118 -0
  109. data/examples/sinatra_app/views/layout.erb +61 -0
  110. data/examples/sinatra_app/views/portfolio.erb +43 -0
  111. data/examples/strategy_generator_example.rb +346 -0
  112. data/hsa_portfolio.csv +11 -0
  113. data/justfile +0 -0
  114. data/lib/api/alpha_vantage_api.rb +462 -0
  115. data/lib/sqa/backtest.rb +329 -0
  116. data/lib/sqa/data_frame/alpha_vantage.rb +43 -65
  117. data/lib/sqa/data_frame/data.rb +92 -0
  118. data/lib/sqa/data_frame/yahoo_finance.rb +35 -43
  119. data/lib/sqa/data_frame.rb +148 -243
  120. data/lib/sqa/ensemble.rb +359 -0
  121. data/lib/sqa/fpop.rb +199 -0
  122. data/lib/sqa/gp.rb +259 -0
  123. data/lib/sqa/indicator.rb +5 -8
  124. data/lib/sqa/init.rb +15 -8
  125. data/lib/sqa/market_regime.rb +240 -0
  126. data/lib/sqa/multi_timeframe.rb +379 -0
  127. data/lib/sqa/pattern_matcher.rb +497 -0
  128. data/lib/sqa/portfolio.rb +260 -6
  129. data/lib/sqa/portfolio_optimizer.rb +377 -0
  130. data/lib/sqa/risk_manager.rb +442 -0
  131. data/lib/sqa/seasonal_analyzer.rb +209 -0
  132. data/lib/sqa/sector_analyzer.rb +300 -0
  133. data/lib/sqa/stock.rb +67 -125
  134. data/lib/sqa/strategy/bollinger_bands.rb +42 -0
  135. data/lib/sqa/strategy/consensus.rb +5 -2
  136. data/lib/sqa/strategy/kbs_strategy.rb +470 -0
  137. data/lib/sqa/strategy/macd.rb +46 -0
  138. data/lib/sqa/strategy/mp.rb +1 -1
  139. data/lib/sqa/strategy/stochastic.rb +60 -0
  140. data/lib/sqa/strategy/volume_breakout.rb +57 -0
  141. data/lib/sqa/strategy.rb +5 -0
  142. data/lib/sqa/strategy_generator.rb +947 -0
  143. data/lib/sqa/stream.rb +361 -0
  144. data/lib/sqa/version.rb +1 -7
  145. data/lib/sqa.rb +23 -16
  146. data/main.just +81 -0
  147. data/mkdocs.yml +288 -0
  148. data/trace.log +0 -0
  149. metadata +261 -51
  150. data/bin/sqa +0 -6
  151. data/lib/patches/dry-cli.rb +0 -228
  152. data/lib/sqa/activity.rb +0 -10
  153. data/lib/sqa/cli.rb +0 -62
  154. data/lib/sqa/commands/analysis.rb +0 -309
  155. data/lib/sqa/commands/base.rb +0 -139
  156. data/lib/sqa/commands/web.rb +0 -199
  157. data/lib/sqa/commands.rb +0 -22
  158. data/lib/sqa/constants.rb +0 -23
  159. data/lib/sqa/indicator/average_true_range.rb +0 -33
  160. data/lib/sqa/indicator/bollinger_bands.rb +0 -28
  161. data/lib/sqa/indicator/candlestick_pattern_recognizer.rb +0 -60
  162. data/lib/sqa/indicator/donchian_channel.rb +0 -29
  163. data/lib/sqa/indicator/double_top_bottom_pattern.rb +0 -34
  164. data/lib/sqa/indicator/elliott_wave_theory.rb +0 -57
  165. data/lib/sqa/indicator/exponential_moving_average.rb +0 -25
  166. data/lib/sqa/indicator/exponential_moving_average_trend.rb +0 -36
  167. data/lib/sqa/indicator/fibonacci_retracement.rb +0 -23
  168. data/lib/sqa/indicator/head_and_shoulders_pattern.rb +0 -26
  169. data/lib/sqa/indicator/market_profile.rb +0 -32
  170. data/lib/sqa/indicator/mean_reversion.rb +0 -37
  171. data/lib/sqa/indicator/momentum.rb +0 -28
  172. data/lib/sqa/indicator/moving_average_convergence_divergence.rb +0 -29
  173. data/lib/sqa/indicator/peaks_and_valleys.rb +0 -29
  174. data/lib/sqa/indicator/predict_next_value.rb +0 -202
  175. data/lib/sqa/indicator/relative_strength_index.rb +0 -47
  176. data/lib/sqa/indicator/simple_moving_average.rb +0 -24
  177. data/lib/sqa/indicator/simple_moving_average_trend.rb +0 -32
  178. data/lib/sqa/indicator/stochastic_oscillator.rb +0 -68
  179. data/lib/sqa/indicator/true_range.rb +0 -39
  180. data/lib/sqa/trade.rb +0 -26
data/docs/data_frame.md CHANGED
@@ -1,164 +1,384 @@
1
- # DataFrame (DF)
1
+ # DataFrame Documentation
2
2
 
3
- A common way to handling data is good. Having multiple ways to import and export datasets is good. Originally SQA::Datastore was intended to provide this functionality but creating that capability took away from the key focs of this project.
3
+ ## Overview
4
4
 
5
- Daru was chosen to fill the gap. The Daru::Vector and Daru::DataFrane classes offer a good abstraction with multiple import and export formats.
5
+ The `SQA::DataFrame` class is a high-performance wrapper around the Polars DataFrame library, specifically designed for time series financial data manipulation. Polars is a Rust-backed library that provides blazingly fast operations on columnar data.
6
6
 
7
- Daru is part of the SciRuby collection. Both Daru and SciRuby are a little out of date.
7
+ ## Architecture
8
8
 
9
- * https://github.com/SciRuby/daru
10
- * https://github.com/SciRuby
9
+ The DataFrame system consists of two main components:
11
10
 
12
- There will be Daru extensions and patches made to adapt it to the specific needs of SQA.
11
+ ### 1. SQA::DataFrame
13
12
 
14
- Frankly, Ruby has lost the battle to Python w/r/t data analysis. The Python equivalent library to Daru is Pandas. It is actively maintained. There is a Ruby gem that uses PyCall to access Pandas but it is a few years out of date with open issues.
13
+ The main DataFrame class that wraps Polars::DataFrame with SQA-specific convenience methods.
15
14
 
16
- I am considering extracting the Daru::DataFrame class into a new gem `sqa-Ddata_frame` so that I can more easily make upgrades and refactor the old thing. It really could use a facelift and a better plugin strategy. The lib/daru/data_frame.rb is over 3,000 lines long. There is a lot of method documentation; but, I not really sure that all of those methods are really needed. We could at least extract each of the methods out into its own file.
15
+ **Location**: `lib/sqa/data_frame.rb`
17
16
 
18
- ## Creating a DataFrame from a CSV File
17
+ **Key Features**:
18
+ - Wraps Polars::DataFrame for high-performance operations
19
+ - Column-based vectorized operations (avoid row iterations)
20
+ - CSV and JSON import/export
21
+ - Automatic column renaming and transformations
22
+ - FPL (Future Period Loss/Profit) analysis convenience methods
23
+ - Method delegation to underlying Polars DataFrame
19
24
 
20
- A common activity is to use financial websites such as https://finance.yahoo.com to download historical price data for a stock.
25
+ ### 2. SQA::DataFrame::Data
21
26
 
22
- Here is how to create a DataFrame from a CSV file downloaded from Finance.yahoo.com ...
27
+ A metadata storage class for stock information, separate from the price/volume data.
28
+
29
+ **Location**: `lib/sqa/data_frame/data.rb`
30
+
31
+ **Attributes**:
32
+ - `ticker` - Stock symbol (e.g., 'AAPL', 'MSFT')
33
+ - `name` - Company name
34
+ - `exchange` - Exchange symbol (NASDAQ, NYSE, etc.)
35
+ - `source` - Data source (`:alpha_vantage`, `:yahoo_finance`)
36
+ - `indicators` - Technical indicators configuration hash
37
+ - `overview` - Company overview data from Alpha Vantage
38
+
39
+ **Key Features**:
40
+ - Dual initialization: from hash (JSON) or keyword arguments
41
+ - JSON serialization with `to_json`
42
+ - Used by `SQA::Stock` to persist metadata in `~/sqa_data/ticker.json`
43
+ - All attributes are read/write accessible
44
+
45
+ ## Creating DataFrames
46
+
47
+ ### From Data Sources
23
48
 
24
49
  ```ruby
25
- df = Daru::DataFrame.from_csv('aapl.csv')
50
+ # Using Alpha Vantage
51
+ stock = SQA::Stock.new(ticker: 'AAPL')
52
+ df = stock.df # SQA::DataFrame instance
26
53
 
27
- # The SQA way uses the file's type to invoke the
28
- # correct method.
29
- df = SQA::DataFrame.load(filename)
54
+ # Using Yahoo Finance
55
+ stock = SQA::Stock.new(ticker: 'MSFT', source: :yahoo_finance)
56
+ df = stock.df
30
57
  ```
31
58
 
32
- The Daru::DataFrame class can be created from many different sources including an ActiveRecord relation -- e.g. you can get you data from a database.
59
+ ### From CSV File
60
+
61
+ ```ruby
62
+ # Load CSV file (Polars-compatible)
63
+ df = SQA::DataFrame.from_csv_file('path/to/stock_data.csv')
33
64
 
34
- ## Using a DataFrame
65
+ # The underlying Polars DataFrame is accessible via .data
66
+ polars_df = df.data
67
+ ```
35
68
 
36
- The column names for a DataFrame are String objects. To get an Array of the column names do this:
69
+ ### From JSON File
37
70
 
38
71
  ```ruby
39
- df.vectors.to_a
40
- #=> ["Date", "Open", "High", "Low", "Close", "Adj Close", "Volume"]
72
+ # Load JSON array of hashes
73
+ df = SQA::DataFrame.from_json_file('path/to/stock_data.json')
41
74
  ```
42
75
 
43
- To get an Array of any specific column do this:
76
+ ### From Array of Hashes
44
77
 
45
78
  ```ruby
46
- df.vectors["The Column Name"].to_a
47
- # where "Any Column Name" could be "Adj Close"
79
+ data = [
80
+ { date: '2024-01-01', close: 150.0, volume: 1_000_000 },
81
+ { date: '2024-01-02', close: 152.0, volume: 1_200_000 }
82
+ ]
48
83
 
84
+ df = SQA::DataFrame.from_aofh(data)
49
85
  ```
50
86
 
51
- You can of course use the `last()` method to constrain your Array to only those entries that make sense during your analysis. Daru::DataFrame supposts both the `first` and `last` methods as well. You can use them to avoid using any more memory in your Array than is needed.
87
+ ## Working with DataFrames
88
+
89
+ ### Column Access
52
90
 
53
91
  ```ruby
54
- df.vectors["The Column Name"].last(14).to_a
55
- # This limits the size of the Array to just the last 14 entries in the DataFrame
92
+ # Get column names
93
+ df.columns
94
+ # => ["timestamp", "open", "high", "low", "close", "adj_close_price", "volume"]
95
+
96
+ # Access a column (returns Polars::Series)
97
+ close_prices = df["close"]
98
+
99
+ # Convert to Ruby array
100
+ prices_array = df["adj_close_price"].to_a
56
101
  ```
57
102
 
58
- ## Renaming the Columns
103
+ ### Basic Operations
104
+
105
+ ```ruby
106
+ # Get dimensions
107
+ df.size # Number of rows (also: df.nrows, df.length)
108
+ df.ncols # Number of columns
109
+
110
+ # Get first/last rows
111
+ df.head(10) # First 10 rows
112
+ df.tail(10) # Last 10 rows
113
+
114
+ # Column statistics (via Polars)
115
+ df.data["close"].mean
116
+ df.data["volume"].sum
117
+ df.data["close"].min
118
+ df.data["close"].max
119
+ ```
59
120
 
60
- You can rename the columns to be symbols. Doing this allows you to use the column names as methods for accessing them in the DataFrame.
121
+ ### Data Transformation
61
122
 
62
123
  ```ruby
63
- old_names = df.vectors.to_a
64
- #=> ["Date", "Open", "High", "Low", "Close", "Adj Close", "Volume"]
124
+ # Rename columns
125
+ mapping = { 'Close' => :close, 'Volume' => :volume }
126
+ df.rename_columns!(mapping)
127
+
128
+ # Apply transformers
129
+ transformers = {
130
+ date: ->(val) { Date.parse(val) },
131
+ volume: ->(val) { val.to_i }
132
+ }
133
+ df.apply_transformers!(transformers)
134
+ ```
65
135
 
66
- new_names = {} # Hash where key is old name, value is new name
67
- #=> {}
136
+ ### Appending DataFrames
68
137
 
69
- df.vectors.each_entry {|old_name| new_names[old_name] = old_name.downcase.gsub(' ','_').to_sym}
138
+ ```ruby
139
+ # Combine two DataFrames
140
+ df1.append!(df2) # Modifies df1 in place
141
+ df1.concat!(df2) # Alias for append!
142
+ ```
143
+
144
+ ### Export
145
+
146
+ ```ruby
147
+ # To CSV
148
+ df.to_csv('output.csv')
149
+ df.write_csv('output.csv') # Alias
70
150
 
71
- new_names
72
- #=>
73
- {"Date"=>:date,
74
- "Open"=>:open,
75
- "High"=>:high,
76
- "Low"=>:low,
77
- "Close"=>:close,
78
- "Adj Close"=>:adj_close,
79
- "Volume"=>:volume}
151
+ # To Hash
152
+ hash = df.to_h
153
+ # => { close: [150.0, 152.0, ...], volume: [1_000_000, 1_200_000, ...] }
154
+ ```
80
155
 
156
+ ## FPL Analysis Methods
81
157
 
82
- df.rename_vectors(new_names)
83
- #=> #<Daru::Index(7): {date, open, high, low, close, adj_close, volume}>
158
+ The DataFrame includes convenience methods for Future Period Loss/Profit analysis:
84
159
 
85
- df.vectors
86
- #=> #<Daru::Index(7): {date, open, high, low, close, adj_close, volume}>
160
+ ### Basic FPL Calculation
87
161
 
88
- # Now you can use the symbolized column name as a method to select that column
89
- df.volume.last(14).volume
90
- #=>
91
- #<Daru::Vector(14)>
92
- volume
93
- 10741 45377800
94
- 10742 37283200
95
- 10743 47471900
96
- 10744 47460200
97
- 10745 48291400
98
- 10746 38824100
99
- 10747 35175100
100
- 10748 50389300
101
- 10749 61235200
102
- 10750 115799700
103
- 10751 97576100
104
- 10752 67823000
105
- 10753 60378500
106
- 10754 54628800
162
+ ```ruby
163
+ # Calculate min/max future deltas for each point
164
+ fpl_data = df.fpl(column: "adj_close_price", fpop: 10)
165
+ # => [[min_delta, max_delta], [min_delta, max_delta], ...]
107
166
  ```
108
167
 
168
+ ### Comprehensive FPL Analysis
109
169
 
170
+ ```ruby
171
+ # Get detailed analysis with risk metrics
172
+ analysis = df.fpl_analysis(column: "adj_close_price", fpop: 10)
173
+ # => [
174
+ # {
175
+ # min_delta: -2.5,
176
+ # max_delta: 5.3,
177
+ # magnitude: 3.9,
178
+ # risk: 7.8,
179
+ # direction: :UP
180
+ # },
181
+ # ...
182
+ # ]
183
+ ```
110
184
 
185
+ See [FPL Analysis Documentation](advanced/fpop.md) for more details.
111
186
 
187
+ ## Working with Stock Metadata
112
188
 
113
- ## Stats on a DataFrame
189
+ ### Creating Metadata
190
+
191
+ ```ruby
192
+ # From keyword arguments
193
+ data = SQA::DataFrame::Data.new(
194
+ ticker: 'AAPL',
195
+ source: :alpha_vantage,
196
+ indicators: { rsi: 14, sma: [20, 50] }
197
+ )
198
+
199
+ # From hash (JSON deserialization)
200
+ json_data = JSON.parse(File.read('aapl.json'))
201
+ data = SQA::DataFrame::Data.new(json_data)
202
+ ```
203
+
204
+ ### Accessing and Modifying Metadata
205
+
206
+ ```ruby
207
+ # Read attributes
208
+ data.ticker # => 'AAPL'
209
+ data.source # => :alpha_vantage
210
+ data.indicators # => { rsi: 14, sma: [20, 50] }
211
+
212
+ # Write attributes
213
+ data.name = 'Apple Inc.'
214
+ data.exchange = 'NASDAQ'
215
+ data.overview = {
216
+ 'company' => 'Apple Inc.',
217
+ 'sector' => 'Technology',
218
+ 'market_cap' => 2_800_000_000_000
219
+ }
220
+ ```
114
221
 
115
- Daru provides some basic tools for the analysis of data stored in a DataFrame. There are too many to cover at this time. Here is a simple example:
222
+ ### Persistence
116
223
 
117
224
  ```ruby
118
- df.last(14)['Adj Close'].minmax
119
- #=> [177.970001, 196.449997]
225
+ # Serialize to JSON
226
+ json_string = data.to_json
120
227
 
121
- # You cab cgabge the order of operations ...
122
- df['Adj Close'].last(14).minmax
123
- #=> [177.970001, 196.449997]
228
+ # Save to file (typically done by SQA::Stock)
229
+ File.write('aapl.json', data.to_json)
124
230
 
125
- # Get a summary report ...
126
- puts df['Adj Close'].last(14).minmax
231
+ # Load from file
232
+ json_data = JSON.parse(File.read('aapl.json'))
233
+ data = SQA::DataFrame::Data.new(json_data)
127
234
  ```
128
- <pre>
129
- = Adj Close
130
- n :14
131
- non-missing:14
132
- median: 192.66500100000002
133
- mean: 188.7521
134
- std.dev.: 7.4488
135
- std.err.: 1.9908
136
- skew: -0.4783
137
- kurtosis: -1.7267
138
- </pre>
139
235
 
140
- ## Bacon in the Sky
236
+ ## Performance Tips
237
+
238
+ 1. **Use Column Operations**: Always prefer Polars column operations over Ruby loops
239
+ ```ruby
240
+ # GOOD: Vectorized operation
241
+ df.data["close"].mean
242
+
243
+ # BAD: Ruby loop
244
+ df["close"].to_a.sum / df.size.to_f
245
+ ```
246
+
247
+ 2. **Access Underlying Polars**: Use `df.data` for direct Polars operations
248
+ ```ruby
249
+ # Direct Polars access
250
+ filtered = df.data.filter(df.data["volume"] > 1_000_000)
251
+ ```
252
+
253
+ 3. **Avoid Unnecessary Array Conversions**: Only convert to arrays when needed
254
+ ```ruby
255
+ # Only convert when passing to external functions
256
+ prices = df["adj_close_price"].to_a
257
+ rsi = SQAI.rsi(prices, period: 14)
258
+ ```
259
+
260
+ 4. **Batch Operations**: Combine operations when possible
261
+ ```ruby
262
+ # Instead of multiple separate operations
263
+ df.apply_transformers!(transformers)
264
+ df.rename_columns!(mapping)
265
+ ```
266
+
267
+ ## Data Sources
268
+
269
+ ### Alpha Vantage
270
+
271
+ **Location**: `lib/sqa/data_frame/alpha_vantage.rb`
141
272
 
142
273
  ```ruby
143
- puts df.ai("when is the best time to buy a stock?")
274
+ SQA::DataFrame::AlphaVantage.recent('AAPL', full: true)
144
275
  ```
145
- The best time to buy a stock is subjective and can vary depending on individual goals, investment strategies, and risk tolerance. However, there are a few general principles to consider:
146
276
 
147
- 1. Valuation: Look for stocks trading at a reasonable or discounted price compared to their intrinsic value. Conduct fundamental analysis to assess a company's financial health, growth prospects, and competitive advantage.
277
+ **Requirements**:
278
+ - Environment variable: `AV_API_KEY` or `ALPHAVANTAGE_API_KEY`
279
+ - Rate limiting: 5 calls/minute (free tier)
148
280
 
149
- 2. Market Timing: Trying to time the market perfectly can be challenging and is often unpredictable. Instead, focus on buying stocks for the long term, considering the company's potential to grow over time.
281
+ ### Yahoo Finance
150
282
 
151
- 3. Diversification: Avoid investing all your funds in a single stock or industry. Diversifying your portfolio across various sectors can help reduce risk and capture different opportunities.
283
+ **Location**: `lib/sqa/data_frame/yahoo_finance.rb`
152
284
 
153
- 4. Patient approach: Practice patience and avoid making impulsive decisions. Regularly monitor the stock's performance, industry trends, and market conditions to make informed choices.
285
+ ```ruby
286
+ SQA::DataFrame::YahooFinance.recent('AAPL', full: true)
287
+ ```
288
+
289
+ **Features**:
290
+ - No API key required
291
+ - Web scraping based (less reliable)
292
+ - Good for testing and fallback
293
+
294
+ ## Adding New Data Sources
154
295
 
155
- 5. Considerations: Take into account any upcoming events that may impact a stock's price, such as earnings announcements, mergers and acquisitions, regulatory changes, or macroeconomic factors.
296
+ To add a new data source adapter:
156
297
 
157
- It's important to note that investing in stocks carries inherent risks, and seeking guidance from financial professionals or conducting thorough research is recommended. Don't ever listen to what an AI has to say about the subject. We are all biased, error prone, and predictability uninformed on the subject.
298
+ 1. Create `lib/sqa/data_frame/my_source.rb`
299
+ 2. Define class `SQA::DataFrame::MySource`
300
+ 3. Implement `self.recent(ticker, **options)` method
301
+ 4. Return data in Polars-compatible format
302
+ 5. Add column mapping if needed
158
303
 
304
+ **Example**:
159
305
 
160
306
  ```ruby
161
- puts df.ai("Yes; but, should I buy this stock now?")
307
+ class SQA::DataFrame::MySource
308
+ TRANSFORMERS = {
309
+ timestamp: ->(val) { Date.parse(val) },
310
+ volume: ->(val) { val.to_i }
311
+ }
312
+
313
+ def self.recent(ticker, **options)
314
+ # Fetch data from your source
315
+ raw_data = fetch_from_api(ticker)
316
+
317
+ # Convert to Polars-compatible format
318
+ SQA::DataFrame.new(raw_data, transformers: TRANSFORMERS)
319
+ end
320
+ end
162
321
  ```
163
- Consulting the magic eight ball cluster.... The future looks cloudy. You should have bought it 14 days ago when I told you it was on its way up! Do you ever listen to me? No! I slave over these numbers night and day. I consult the best magic eight ball sources available. What do I get for my efforts? Nothing!
164
322
 
323
+ ## Common Gotchas
324
+
325
+ 1. **DataFrame vs Polars**:
326
+ - `df` is `SQA::DataFrame`
327
+ - `df.data` is `Polars::DataFrame`
328
+
329
+ 2. **Column Names**:
330
+ - Column names are strings, not symbols
331
+ - Use `df["close"]` not `df[:close]`
332
+
333
+ 3. **Method Delegation**:
334
+ - Unknown methods are delegated to Polars::DataFrame
335
+ - Check Polars docs for advanced operations
336
+
337
+ 4. **Indicators Need Arrays**:
338
+ - Extract data with `.to_a` before passing to SQAI/TAI functions
339
+ - Example: `prices = df["close"].to_a`
340
+
341
+ ## Related Documentation
342
+
343
+ - [FPL Analysis](advanced/fpop.md) - Future Period Loss/Profit utilities
344
+ - [Stock Class](api/stock.md) - Using DataFrames with Stock objects
345
+ - [Indicators](indicators/index.md) - Technical indicator integration
346
+ - [Polars Documentation](https://pola-rs.github.io/polars-book/) - Underlying library
347
+
348
+ ## Example: Complete Workflow
349
+
350
+ ```ruby
351
+ require 'sqa'
352
+
353
+ SQA.init
354
+
355
+ # Load stock data
356
+ stock = SQA::Stock.new(ticker: 'AAPL')
357
+
358
+ # Access DataFrame
359
+ df = stock.df
360
+
361
+ # Get price array for indicators
362
+ prices = df["adj_close_price"].to_a
363
+
364
+ # Calculate technical indicators
365
+ sma_20 = SQAI.sma(prices, period: 20)
366
+ rsi_14 = SQAI.rsi(prices, period: 14)
367
+
368
+ # FPL analysis
369
+ fpl_analysis = df.fpl_analysis(fpop: 10)
370
+ high_quality = SQA::FPOP.filter_by_quality(
371
+ fpl_analysis,
372
+ min_magnitude: 5.0,
373
+ max_risk: 25.0
374
+ )
375
+
376
+ # Access stock metadata
377
+ puts "Ticker: #{stock.ticker}"
378
+ puts "Exchange: #{stock.exchange}"
379
+ puts "Source: #{stock.source}"
380
+
381
+ # Export data
382
+ df.to_csv("aapl_prices.csv")
383
+ File.write("aapl_metadata.json", stock.data.to_json)
384
+ ```
@@ -0,0 +1,26 @@
1
+ ## Factors that Impact Price
2
+
3
+ By considering these factors, investors can better understand the potential price movements of stocks and make informed investment decisions.
4
+
5
+ This breakdown approximates how heavily each factor might typically weigh on stock prices across a varied portfolio. However, in specific scenarios, the impact of each factor could be dramatically different. For instance, in times of global financial crisis, economic indicators and market sentiment might dramatically outweigh company-specific performance, or during a technological breakthrough, industry trends could be the dominant factor.
6
+
7
+ Ranked most impactful to least impactful.
8
+
9
+ 1. **Company Performance**: 35%
10
+ - The financial health and growth prospects of a company are crucial. Positive earnings reports, strong revenue growth, and effective management often directly translate to increased stock prices.
11
+
12
+ 2. **Market Sentiment**: 25%
13
+ - This is a broad factor that encompasses the psychological aspects of the market. Positive market sentiment can drive prices up rapidly, sometimes irrespective of underlying fundamentals.
14
+
15
+ 3. **Economic Indicators**: 15%
16
+ - Broad economic trends greatly influence investor confidence and expectations. Robust economic growth, for example, tends to buoy most stocks, particularly in sectors closely tied to economic cycles.
17
+
18
+ 4. **Interest Rates**: 10%
19
+ - Changes in interest rates by central banks can shift investor preferences between stocks and bonds, thus having a significant impact on stock prices.
20
+
21
+ 5. **Geopolitical Events**: 10%
22
+ - These events can create considerable market volatility and can impact specific sectors more intensely (e.g., energy stocks during oil crises).
23
+
24
+ 6. **Industry Trends**: 5%
25
+ - While these can be highly impactful on specific stocks or sectors, their broader impact on stock prices across the market tends to be more contained compared to the other factors unless the trend sparks a sector-wide shift.
26
+
data/docs/finviz.md ADDED
@@ -0,0 +1,11 @@
1
+ # Financial Vizualization (finviz)
2
+
3
+ The original finviz unofficial gem has been untouched for 3 years. I just forked it in the hope that I can bring it forward to current versions of its dependencies.
4
+
5
+ https;//finviz.com
6
+
7
+ ## AAPL weekly
8
+
9
+ The default is daily.
10
+
11
+ https://finviz.com/quote.ashx?t=AAPL&p=w
@@ -0,0 +1,25 @@
1
+ # FXProBot
2
+
3
+ FXProBot is a new trading software by Avenix Fzco, a major fintech firm located in Dubai that specializes in automated trading. This advanced robot has been built with the sole aim of improving the experience of both established and novice traders by using sophisticated algorithms and strong risk management approaches. The goal of FXProBot is to change how investors interact with the market; it offers an efficient and effortless platform for forex traders.
4
+
5
+ ## Main Features of FXProBot
6
+
7
+ Impulse Detection and Analysis – One of the key attributes of this bot is its well-crafted impulse detection and analysis process. Before initiating any trade, this forex robot scans market impulses, searching for ones that may benefit traders. In instances such as when Doji candles appear or ambiguous market behavior occurs, this feature helps avoid common mistakes.
8
+
9
+ **Trend Conformity** – This feature comes into play when market impulses are aligned with broader market trends. Trade entry accuracy is enhanced by FXProBot’s built-in indicators. In this way, it increases the chances for better trading opportunities, since FXProBot trades in line with prevailing market trends.
10
+
11
+ **Robust Risk Management** – Successful trading necessitates good risk management, which FXProBot demonstrates excellently. Each trade executed by this forex robot incorporates predefined levels for Stop Loss (SL) and Take Profit (TP). This strong framework used for risk management promotes minimal losses, thus safeguarding profits and investments from instability.
12
+
13
+ **Automated Trading** – Trading has become simpler thanks to automated trading offered by FXProBot. This Expert Advisor (EA) uses complex algorithms, ensuring accuracy as it removes emotions and psychological stress from trading decisions. This means that traders can focus on their strategies while FXProBot takes care of execution.
14
+
15
+ **Customizable Settings** – In order to accommodate different trading styles, FXProBot is highly customizable. This allows users to modify the EA’s risk level, trade sizes, and select between numerous indicators available, thereby fine-tuning it to fit any individual preferences.
16
+
17
+ **Backtesting and Optimization** – Backtesting as well as optimization are offered by FXProBot. Traders are able to put their strategies to the test through extensive historical data prior to applying them in live markets. These techniques help improve how trades should be executed, thereby increasing the chances of success.
18
+
19
+ **Regular Updates and Comprehensive Support** – Avenix Fzco constantly aims at improving its creations; FXProBot is regularly updated with market insights and new algorithms for better strategy applications towards changing market conditions. Also, within all stages of one’s trading journey with FXProBot, he or she will have a dedicated support team right from installation up to troubleshooting.
20
+
21
+ **Educational Resources** – In addition, FXProBot provides a wide range of educational resources such as tutorials, webinars, and articles on basic forex concepts and advanced trading methods. These resources enable one to make informed decisions while participating in forex-related activities.
22
+
23
+ ## About Avenix Fzco
24
+
25
+ Avenix Fzco, a fintech pioneer in Dubai, UAE, is focused on creating high-level trading software for forex traders. The company’s most recent release, FXProBot, shows its dedication to precision, risk management, and intelligent operations. For more details about FXProBot and how it can improve the trading experience, please visit https://fxprobot.com/.
@@ -0,0 +1,104 @@
1
+ ### Overview of Genetic Programming (GP)
2
+
3
+ Genetic Programming (GP) is an evolutionary algorithm-inspired methodology used to evolve programs or models, particularly to solve optimization and search problems. GP simulates the process of natural selection by treating potential solutions as members of a population of programs. These programs are encoded as trees, where the nodes represent operations or functions and the leaves represent input data. During the GP process, populations evolve over generations using genetic operations such as mutation, crossover (recombination), and selection according to their fitness.
4
+
5
+ #### definitions
6
+ ##### Single Signal with Varying Parameters
7
+
8
+ prices: historical price data for a stock (Array of Hash)
9
+ pww: past window width (Integer)
10
+ fww: future window width (Integer)
11
+ gain: (Float)
12
+ loss: (Float)
13
+ indicator: historical signal values (Float)
14
+ signal: buy/sell/hold (Symbol)
15
+ x: today - instance in time (Integer)
16
+
17
+ Different fitness functions for buy/sell/hold
18
+
19
+ pwi = (x-pww .. x)
20
+ fwi = (x+1 .. x+fww)
21
+
22
+ signal = signal_function(prices, x, pww)
23
+ gain = prices[fwi].max - prices[x] # could be negative
24
+ loss = prices[fwi].min - prices[x]
25
+ delta = decision-point amount (Float)
26
+ buy when forcast_price - prices[x] >= delta
27
+ buy to realize gain
28
+
29
+ sell when prices[x] - forcast_price >= delta
30
+ sell to avoid loss
31
+
32
+ fitness_sell = gain - delta # more positive the better
33
+ fitness_buy = loss + delta # more negative the better
34
+ fitness_hold = gain.abs < delta && loss.abs < delta
35
+
36
+ Want to mutate pww, fww -- maybe x? to see if solution holds
37
+
38
+ ##### Multiple Signals with varying parameters
39
+
40
+ TBD
41
+
42
+ ### How GP Can Be Applied to Stock Market Predictions
43
+
44
+ In the domain of stock market predictions, GP can be particularly useful for evolving strategies to make decisions such as buy, sell, or hold based on historical data. Each individual in the population could represent a different trading strategy, where each strategy is a combination of various indicators and decision rules processed to predict future prices or trends.
45
+
46
+ The fitness of each program is measured by how well it predicts the stock price movement and could be based on factors such and minimization of loss or maximization of gains during simulated trading.
47
+
48
+ ### Example Application: Genetic Programming in Ruby
49
+
50
+ This is a simplified example using the `darwinning` gem for applying a genetic programming approach to determine the best strategy among signal indicators for stock trading. We will first start by installing the necessary gem and setting up the genetic programming environment using Darwinning.
51
+
52
+ 1. **Installation**: You'll need to install the Darwinning gem first. You can do this using:
53
+
54
+ ```bash
55
+ gem install darwinning
56
+ ```
57
+
58
+ 2. **Designing the Program**: Assume that our trading decision is based on a simplistic model using standard indicators like moving averages (MA), with parameters evolving to fit historical data for maximum gains.
59
+
60
+ Here is the Ruby code implementing genetic programming using Darwinning:
61
+
62
+ ```ruby
63
+ require 'darwinning'
64
+
65
+ class TradingStrategy < Darwinning::Organism
66
+ @name = "Trading Strategy"
67
+ @genes = [
68
+ Darwinning::Gene.new(name: "moving average short period", value_range: (5..15)),
69
+ Darwinning::Gene.new(name: "moving average long period", value_range: (20..50)),
70
+ Darwinning::Gene.new(name: "buy threshold", value_range: (0.01..0.05)),
71
+ Darwinning::Gene.new(name: "sell threshold", value_range: (-0.05..-0.01))
72
+ ]
73
+
74
+ def fitness
75
+ # Define how to evaluate the strategy's fitness. Here, simplistic measures could be the paper trading results:
76
+ moving_average_short = genotypes["moving average short period"]
77
+ moving_accurate_touch_trigger_index_Bigvar= >umbrella_deal; share_rally_hook="?pelicans" surpassing_value_trauma_long = {5000..100000}
78
+ # Simplified fitness function calculation for demonstration: assume returns as random for illustration
79
+ rand(100) - 50 # Random fitness value for demonstration
80
+ end
81
+ end
82
+
83
+ population = Darwinning::Population.new(
84
+ organism: TradingStrategy, population_size: 30,
85
+ fitness_goal: 100, generations_limit: 100
86
+ )
87
+
88
+ population.evolve!
89
+
90
+ puts "Best trading strategy: #{population.best_member}"
91
+ ```
92
+
93
+ ### Considerations
94
+
95
+ - **Data and Fitness Function**: The quality and relevance of input data are critical. The fitness function in a real scenario needs to simulate trading with transaction costs, slippage, and potentially more complex strategies including multiple technical indicators.
96
+ - **Program Representation**: We represented trading strategies using genes corresponding to their strategy parameters.
97
+ - **Evaluation**: It's important to evaluate your model on unseen data to ensure it generalizes well and doesn't overfit the historical data.
98
+
99
+ ### Conclusion
100
+
101
+ Utilizing genetic programming with the Ruby `darwinning` gem in financial settings requires careful consideration of genetic representation, fitness function design, and evaluation methodologies. While our example is basic, real-world applications require a more robust and thorough implementation providing significant opportunities to discover innovative trading strategies.
102
+
103
+ Despite the basic example, real-world financial applications would need a much more robust and thorough implementation considering various factors like transaction costs, slippage, and more sophisticated financial metrics for performance evaluations. Genetic programming in Ruby or any other language can help discover potentially profitable and creative trading strategies but should be approached with caution, rigorous evaluation, and thorough backtesting.
104
+