sqa 0.0.24 → 0.0.31

data/develop_summary.md

@@ -0,0 +1,313 @@
+Comprehensive Summary: Develop Branch Changes
+Overview
+
+The develop branch contains 12 commits spanning from February 2024 to August 2025 (18+ months of development). This represents a major evolution of the SQA project with significant architectural changes and new features.
+
+Impact: 104,907 insertions, 2,017 deletions across 47 files
+🎯 Strategic Direction Change
+From Application to Library
+
+Commit: 11351a6 (Feb 2024) - "refocused effort on a library rather than library plus application"
+
+Major Architectural Shift:
+
+    Removed entire CLI command structure:
+        Deleted lib/sqa/cli.rb (62 lines)
+        Deleted lib/sqa/commands.rb (22 lines)
+        Deleted lib/sqa/commands/analysis.rb (309 lines)
+        Deleted lib/sqa/commands/base.rb (139 lines)
+        Deleted lib/sqa/commands/web.rb (199 lines)
+        Deleted lib/sqa/constants.rb (23 lines)
+        Removed dry-cli patches (228 lines)
+
+    Removed configuration testing infrastructure:
+        All config test files deleted
+        CLI override testing removed
+
+Rationale: Simplification - focus on providing a reusable library for quantitative analysis rather than a full application with CLI.
+🔧 Core Technical Improvements
+1. DataFrame Refactoring (Major Change)
+
+Commit: 9625e31 (Feb 2025) - "Refactor SQA::DataFrame to Integrate with polars-df Gem"
+
+Before:
+
+# Based on Hashie::Mash wrapper
+class Data < Hashie::Mash
+  # Custom wrapper around hash-based data structure
+end
+
+After:
+
+# Based on Polars (Rust-backed high-performance dataframe)
+def initialize(raw_data = nil, mapping: {}, transformers: {})
+  @data = Polars::DataFrame.new(raw_data)
+end
+
+Benefits:
+
+    Blazingly fast - Polars is written in Rust
+    Industry-standard DataFrame API (similar to Python's pandas/Polars)
+    Better memory efficiency
+    Native support for complex data operations
+    84 insertions, 272 deletions (net reduction + improved functionality)
+
+2. Custom Alpha Vantage API Wrapper
+
+Commits: 162dbdc, 9fb7727 (Feb 2024)
+
+New File: lib/api/alpha_vantage_api.rb (462 lines)
+
+Rationale: Replaced the alphavantage gem dependency with custom implementation
+
+Features:
+
+    Direct Faraday-based HTTP client
+    Comprehensive API coverage (70+ methods)
+    Categories covered:
+        Market Data (quotes, intraday, daily, etc.)
+        Technical Indicators (AD, ADOSC, ADX, AROON, etc.)
+        Economic Indicators (commodities, aluminum, etc.)
+        Fundamental Data (balance sheet, earnings, etc.)
+        Digital & Forex markets
+    Better error handling
+    More maintainable and customizable
+
+Example Methods:
+
+def adx(symbol:, interval:, time_period:)
+def balance_sheet(symbol:)
+def aroon(symbol:, interval:, time_period:)
+
+3. Genetic Programming for Strategy Evaluation
+
+Commits: 578032b, 84b8ff5 (Oct 2024) - "SQA-144 Add GeneticProgram class for strategy evaluation"
+
+New Files:
+
+    lib/sqa/gp.rb (107 lines) - Genetic Programming implementation
+    docs/genetic_programming.md (104 lines) - Comprehensive documentation
+
+Purpose: Use evolutionary algorithms to discover optimal trading strategies
+
+Key Concepts:
+
+# Fitness functions for trading signals
+fitness_sell = gain - delta  # more positive = better
+fitness_buy  = loss + delta   # more negative = better
+fitness_hold = gain.abs < delta && loss.abs < delta
+
+# Mutation constraints
+constraint = {
+  indicators: [:rsi, :ma, :raw, :whatever],
+  pww:  (5..15).to_a,    # past window width
+  fww:  (5..15).to_a,    # future window width
+  x:    (-30..-15).to_a  # time instance
+}
+
+Integration: Uses the darwinning gem for genetic algorithm implementation
+4. Black-Scholes Options Pricing
+
+New File: lib/sqa/indicator/black_scholes.rb (61 lines)
+
+Implementation:
+
+def black_scholes(
+  option_type,          # 'c' for call, 'p' for put
+  current_stock_price,
+  strike_price,
+  time_to_expiration,   # in years
+  risk_free_rate,       # as decimal
+  volatility            # as decimal
+)
+
+Features:
+
+    European call and put option pricing
+    Normal distribution CDF implementation
+    Well-documented with volatility calculation examples
+    Meaningful parameter names (refactored from cryptic originals)
+
+📚 Documentation Expansion
+
+New Documentation Files:
+
+    docs/genetic_programming.md (104 lines)
+        Overview of GP methodology
+        Application to stock market predictions
+        Ruby implementation examples with darwinning gem
+        Fitness function design
+        Backtesting considerations
+
+    docs/factors_that_impact_price.md (26 lines)
+        Market factors affecting stock prices
+
+    docs/options.md (8 lines)
+        Options trading basics
+
+    docs/fx_pro_bit.md (25 lines)
+        Foreign exchange and cryptocurrency notes
+
+    docs/finviz.md (11 lines)
+        Note about finviz.com website for market screening
+
+    docs/i_gotta_an_idea.md (22 lines)
+        Project ideas and future directions
+
+    COMMITS.md (196 lines)
+        Conventional Commits specification (latest commit Aug 2025)
+        Standardized commit message format
+        Semantic versioning alignment
+
+🔄 Dependency Updates
+Added Dependencies (Gemfile)
+
+gem 'csv'        # CSV Reading and Writing
+gem 'eps'        # Machine learning for Ruby
+gem 'polars-df'  # Blazingly fast DataFrames
+gem 'raix'       # Ruby AI eXtensions
+gem 'regent'     # AI Agents in Ruby
+gem 'rspec'      # Testing framework
+gem 'toml-rb'    # TOML parser
+
+Gemspec Changes
+
+# Removed
+- 'dry-cli'           # CLI framework no longer needed
+- 'sem_version'       # Replaced with versionaire
+
+# Added
++ 'ruby_llm'          # LLM integration
++ 'ruby_llm-mcp'      # LLM MCP protocol
++ 'shared_tools'      # Shared utilities
+
+# Updated
+~ 'hashie', '~>4.1.0' => 'hashie' # Version constraint relaxed
+~ spec.required_ruby_version = ">= 2.7" => ">= 3.2"  # Ruby 3.2+ required
+
+Note: Ruby version requirement increased from 2.7+ to 3.2+
+🗂️ Data & Configuration
+New Data Files
+
+    data/talk_talk.json (103,284 lines)
+        Massive dataset addition
+        Likely stock price or trading data
+        Represents 98.5% of all line additions
+
+    hsa_portfolio.csv (11 lines)
+        Sample HSA (Health Savings Account) portfolio data
+
+    .gitmessage.txt (5 lines)
+        Git commit message template
+
+    .semver (6 lines)
+        Semantic versioning configuration
+
+Build Automation
+
+main.just (81 lines)
+
+    Justfile for task automation
+    Modules for repo, gem, version, git operations
+    Tasks: install, test, coverage, flay (static analysis)
+    Man page generation with kramdown-man
+
+🧪 Testing Improvements
+
+Commits: ee314d6 (Feb 2024) - "corrected some typos; filled out skipped data_frame_test cases"
+
+Changes to test/data_frame_test.rb:
+
+    46 insertions, 242 deletions (net simplification)
+    Previously skipped test cases now implemented
+    Found errors in DataFrame requiring research
+    Tests updated for Polars-based DataFrame
+
+📊 Version History
+
+Version Bump: 0.0.25 → 0.0.31 (6 minor versions)
+
+Version System Changes:
+
+    Removed SQA.version class method
+    Simplified to constant: SQA::VERSION = '0.0.31'
+    Planning to replace semver gem with versionaire
+
+🤖 AI/ML Integration Direction
+
+New AI-Related Dependencies:
+
+    eps - Machine learning (regression & classification)
+    raix - Ruby AI extensions
+    regent - AI Agents library
+    ruby_llm - LLM integration
+    ruby_llm-mcp - LLM MCP protocol support
+
+Genetic Programming:
+
+    Custom GP implementation for strategy discovery
+    Darwinning gem integration
+    Automated trading strategy evolution
+
+Direction: The project is evolving toward AI-assisted quantitative trading strategy development
+📝 Notable Files Modified
+Stock.rb Refactoring
+
+    38 insertions, 127 deletions (net simplification)
+    Likely adapted to new DataFrame structure
+
+Data Source Adapters
+
+    lib/sqa/data_frame/alpha_vantage.rb - Simplified (18 ins, 52 del)
+    lib/sqa/data_frame/yahoo_finance.rb - Refactored (34 ins, 43 del)
+    Both adapted to Polars DataFrame
+
+🎯 Timeline Summary
+
+| Date | Focus Area | Key Achievement | |------|-----------|----------------| | Feb 2024 | Architecture | Removed CLI, created Alpha Vantage API wrapper | | Feb 2024 | Testing | Filled DataFrame test gaps | | May 2024 | Versioning | Planned semver → versionaire migration | | Jun 2024 | Documentation | Added market analysis docs | | Oct 2024 | AI/ML | Added Genetic Programming for strategies | | Dec 2024 | Research | Documented finviz.com resource | | Feb 2025 | Performance | Major: Migrated to Polars DataFrame | | Feb 2025 | Dependencies | Added ML/AI gems (eps, raix, regent, ruby_llm) | | Aug 2025 | Standards | Adopted Conventional Commits |
+⚠️ Breaking Changes
+
+    Ruby Version: Now requires Ruby 3.2+ (was 2.7+)
+    API Changes: DataFrame completely refactored (Hashie → Polars)
+    No CLI: Entire command-line interface removed
+    Dependencies: Several removed (dry-cli, sem_version)
+    Configuration: Test config files removed
+
+✅ Merge Readiness Assessment
+Strengths
+
+    Well-tested DataFrame refactoring
+    Comprehensive documentation
+    Clear architectural direction
+    Modern dependencies
+    Performance improvements with Polars
+
+Concerns
+
+    Massive data file (103k line JSON) - may need .gitignore
+    Some commits mention finding errors needing research
+    Breaking changes will affect existing users
+    Ruby 3.2+ requirement may exclude some users
+
+Recommendation
+
+Review before merging:
+
+    Verify all DataFrame tests pass
+    Consider excluding data/talk_talk.json from version control
+    Document breaking changes in CHANGELOG
+    Consider semantic versioning: this should be 1.0.0 (major rewrite)
+    Ensure AI/ML features are production-ready
+
+🚀 Future Direction
+
+The develop branch shows SQA evolving from a simple CLI tool into a comprehensive AI-powered quantitative trading library with:
+
+    High-performance data processing (Polars)
+    Genetic algorithm strategy discovery
+    Options pricing (Black-Scholes)
+    LLM integration capabilities
+    Machine learning models
+    AI agent support
+
+This is a significant and ambitious evolution of the project.

data/docs/advanced/backtesting.md

@@ -0,0 +1,206 @@
+# Backtesting Framework
+
+## Overview
+
+Simulate trading strategies on historical data to evaluate performance before risking real capital.
+
+## Quick Start
+
+```ruby
+require 'sqa'
+
+# Load stock data
+stock = SQA::Stock.new(ticker: 'AAPL')
+
+# Create backtest
+backtest = SQA::Backtest.new(
+  stock: stock,
+  strategy: SQA::Strategy::RSI,
+  initial_cash: 10_000,
+  commission: 1.0
+)
+
+# Run simulation
+results = backtest.run
+
+# View results
+puts "Total Return: #{results.total_return}%"
+puts "Sharpe Ratio: #{results.sharpe_ratio}"
+puts "Max Drawdown: #{results.max_drawdown}%"
+puts "Win Rate: #{results.win_rate}%"
+```
+
+## Configuration
+
+### Basic Parameters
+
+```ruby
+backtest = SQA::Backtest.new(
+  stock: stock,                    # SQA::Stock object
+  strategy: SQA::Strategy::MACD,   # Strategy class
+  initial_cash: 10_000,            # Starting capital
+  commission: 1.0,                 # Per-trade commission
+  position_size: 100               # Shares per trade
+)
+```
+
+## Results Object
+
+The `backtest.run` method returns a results object with:
+
+### Return Metrics
+- `total_return` - Total percentage return
+- `annualized_return` - Return per year
+- `benchmark_return` - Buy-and-hold comparison
+
+### Risk Metrics
+- `sharpe_ratio` - Risk-adjusted return
+- `sortino_ratio` - Downside risk-adjusted return
+- `max_drawdown` - Largest peak-to-trough decline
+- `volatility` - Standard deviation of returns
+
+### Trading Metrics
+- `num_trades` - Total number of trades
+- `win_rate` - Percentage of profitable trades
+- `avg_win` - Average winning trade
+- `avg_loss` - Average losing trade
+- `profit_factor` - Gross profit / gross loss
+
+### Portfolio Metrics
+- `final_value` - Ending portfolio value
+- `portfolio_value` - Value over time (array)
+- `trades` - Full trade history
+
+## Complete Example
+
+```ruby
+# Load data
+stock = SQA::Stock.new(ticker: 'AAPL')
+
+# Test multiple strategies
+strategies = [
+  SQA::Strategy::RSI,
+  SQA::Strategy::MACD,
+  SQA::Strategy::BollingerBands
+]
+
+strategies.each do |strategy|
+  backtest = SQA::Backtest.new(
+    stock: stock,
+    strategy: strategy,
+    initial_cash: 10_000
+  )
+  
+  results = backtest.run
+  
+  puts "\n#{strategy.name} Results:"
+  puts "-" * 40
+  puts "Return: #{results.total_return.round(2)}%"
+  puts "Sharpe: #{results.sharpe_ratio.round(2)}"
+  puts "Max DD: #{results.max_drawdown.round(2)}%"
+  puts "Trades: #{results.num_trades}"
+  puts "Win Rate: #{results.win_rate.round(2)}%"
+end
+```
+
+## Advanced Usage
+
+### Custom Date Range
+
+```ruby
+# Backtest specific period
+stock = SQA::Stock.new(
+  ticker: 'AAPL',
+  start_date: '2020-01-01',
+  end_date: '2023-12-31'
+)
+
+backtest = SQA::Backtest.new(stock: stock, strategy: strategy)
+results = backtest.run
+```
+
+### Parameter Optimization
+
+```ruby
+# Test different RSI thresholds
+(20..40).step(5).each do |oversold|
+  results = test_strategy_with_params(oversold)
+  puts "RSI #{oversold}: Return = #{results.total_return}%"
+end
+```
+
+### Walk-Forward Testing
+
+```ruby
+# Train on first 70%, test on last 30%
+train_end = (stock.df.height * 0.7).to_i
+train_stock = stock.df[0...train_end]
+test_stock = stock.df[train_end..-1]
+
+# Optimize on training data
+best_params = optimize_on(train_stock)
+
+# Validate on test data
+results = backtest_with_params(test_stock, best_params)
+```
+
+## Visualization
+
+```ruby
+# Plot equity curve
+require 'gruff'
+
+g = Gruff::Line.new
+g.title = "Portfolio Value Over Time"
+g.data("Portfolio", results.portfolio_value)
+g.write('equity_curve.png')
+```
+
+## Best Practices
+
+### 1. Use Enough Data
+- Minimum: 252 trading days (1 year)
+- Recommended: 1260+ days (5 years)
+- Include different market conditions
+
+### 2. Account for Costs
+```ruby
+backtest = SQA::Backtest.new(
+  commission: 1.0,        # Per trade
+  slippage: 0.001         # 0.1% slippage
+)
+```
+
+### 3. Avoid Overfitting
+- Use walk-forward analysis
+- Test on out-of-sample data
+- Limit parameter optimization
+
+### 4. Compare to Benchmark
+```ruby
+if results.total_return < results.benchmark_return
+  puts "⚠️  Strategy underperforms buy-and-hold"
+end
+```
+
+### 5. Check Multiple Metrics
+Don't rely on return alone:
+- Sharpe ratio (risk-adjusted)
+- Max drawdown (worst case)
+- Win rate (consistency)
+- Profit factor (quality)
+
+## Common Pitfalls
+
+❌ **Survivorship Bias**: Only testing stocks that still exist  
+❌ **Look-Ahead Bias**: Using future data  
+❌ **Curve Fitting**: Over-optimizing parameters  
+❌ **Ignoring Costs**: Unrealistic commission/slippage  
+❌ **Small Sample**: Too few trades for statistical significance  
+
+## Related
+
+- [Portfolio Management](portfolio.md) - Track positions and P&L
+- [Strategy Generator](strategy-generator.md) - Discover patterns
+- [Risk Management](risk-management.md) - Position sizing and risk metrics
+

data/docs/advanced/ensemble.md

@@ -0,0 +1,68 @@
+# Ensemble Strategies
+
+Combine multiple strategies with voting and meta-learning.
+
+## Basic Ensemble
+
+```ruby
+ensemble = SQA::Ensemble.new(
+  strategies: [
+    SQA::Strategy::RSI,
+    SQA::Strategy::MACD,
+    SQA::Strategy::BollingerBands
+  ],
+  voting_method: :majority
+)
+
+signal = ensemble.signal(vector)  # => :buy, :sell, or :hold
+```
+
+## Voting Methods
+
+### Majority Voting
+```ruby
+voting_method: :majority  # Most common signal wins
+```
+
+### Weighted Voting
+```ruby
+voting_method: :weighted
+# Weight strategies by past performance
+```
+
+### Unanimous
+```ruby
+voting_method: :unanimous  # All must agree
+```
+
+### Confidence-Based
+```ruby
+voting_method: :confidence  # Weight by confidence scores
+```
+
+## Dynamic Weighting
+
+```ruby
+# Update weights based on performance
+ensemble.update_weight(SQA::Strategy::RSI, 1.5)   # Increase weight
+ensemble.update_weight(SQA::Strategy::MACD, 0.5)  # Decrease weight
+```
+
+## Strategy Rotation
+
+```ruby
+# Select best strategy for current market
+selected = ensemble.rotate(stock)
+# Automatically switches based on recent performance
+```
+
+## Backtest Comparison
+
+```ruby
+comparison = ensemble.backtest_comparison(stock)
+
+comparison.each do |name, results|
+  puts "#{name}: Return = #{results[:return]}%"
+end
+```
+