sqa 0.0.22 → 0.0.31

data/README.md

@@ -1,353 +1,973 @@
 # SQA - Simple Qualitative Analysis
 
-This is a very simplistic set of tools for running technical analysis (quantitative and qualitative) on a stock portfolio.  Simplistic means it is not reliable nor intended for any kind of mission-critical financial use.  Think of it as a training tool.  I do.  Its helping me understand why I need professional help from people who know what they are doing.
+[![Badges?](https://img.shields.io/badge/Badge-We%20don't%20need%20no%20stinkin'%20badges!-red)](https://www.youtube.com/watch?v=VqomZQMZQCQ)
 
-The BUY/SELL signals that it generates should not be taken seriously.  **DO NOT USE** this library when real money is at stake.  If you lose your shirt playing in the stock market don't come crying to me.  I think playing in the market is like playing in the street.  You are going to get run over.
+A Ruby library for technical analysis of stock market data. This is a simplistic set of tools for quantitative and qualitative stock analysis designed for **educational purposes only**.
+
+**⚠️ WARNING:** This is a learning tool, not production software. Do NOT use this library when real money is at stake. The BUY/SELL signals it generates should not be taken seriously. If you lose your shirt playing in the stock market, don't come crying to me. Playing in the market is like playing in the street - you're going to get run over.
 
 <!-- Tocer[start]: Auto-generated, don't remove. -->
 
 ## Table of Contents
 
-  - [This is a Work in Progress](#this-is-a-work-in-progress)
+  - [Features](#features)
   - [Installation](#installation)
-  - [Semantic Versioning](#semantic-versioning)
+  - [Configuration](#configuration)
+    - [Data Directory](#data-directory)
+    - [Alpha Vantage API Key](#alpha-vantage-api-key)
   - [Usage](#usage)
-    - [Setup a Config File](#setup-a-config-file)
-    - [AlphaVantage](#alphavantage)
-    - [TradingView](#tradingview)
+    - [Working with Stocks](#working-with-stocks)
+    - [DataFrame Operations](#dataframe-operations)
+    - [Technical Indicators](#technical-indicators)
+    - [Trading Strategies](#trading-strategies)
+    - [Statistics](#statistics)
+  - [Interactive Console](#interactive-console)
+  - [Architecture](#architecture)
+  - [Data Sources](#data-sources)
+    - [Alpha Vantage](#alpha-vantage)
     - [Yahoo Finance](#yahoo-finance)
-  - [Playing in IRB](#playing-in-irb)
-    - [With Stocks and Indicators](#with-stocks-and-indicators)
-      - [Get Historical Prices](#get-historical-prices)
-    - [Playing in the IRB - Setup](#playing-in-the-irb---setup)
-    - [Playing in the IRB - Statistics](#playing-in-the-irb---statistics)
-    - [Playing in the IRB - Indicators](#playing-in-the-irb---indicators)
-    - [Playing in the IRB - Strategies](#playing-in-the-irb---strategies)
-  - [Included Program Examples](#included-program-examples)
-    - [Analysis](#analysis)
-    - [Web](#web)
   - [Contributing](#contributing)
   - [License](#license)
 
 <!-- Tocer[finish]: Auto-generated, don't remove. -->
 
-## This is a Work in Progress
+## Features
+
+### Core Capabilities
+- **High-Performance DataFrames** - Polars-based data structures for time series financial data
+- **150+ Technical Indicators** - Via the `sqa-tai` gem (TA-Lib wrapper)
+- **Trading Strategies** - Framework for building and testing trading strategies
+- **Multiple Data Sources** - Alpha Vantage and Yahoo Finance adapters
+- **Stock Management** - Track stocks with historical prices and company metadata
+- **Statistical Analysis** - Comprehensive statistics on price data
+- **Ticker Validation** - Validate stock symbols against market exchanges
+- **Interactive Console** - IRB console for experimentation (`sqa-console`)
+
+### Advanced Features
+- **Portfolio Management** - Track positions, trades, P&L with commission support
+- **Backtesting Framework** - Simulate trading strategies with comprehensive performance metrics
+- **Strategy Generator** - Reverse-engineer profitable trades to discover indicator patterns
+- **Genetic Programming** - Evolve optimal strategy parameters through natural selection
+- **Knowledge-Based Strategy (KBS)** - RETE-based forward-chaining inference for complex trading rules
+- **Real-Time Streaming** - Event-driven live price processing with on-the-fly signal generation
+- **FPL Analysis** - Future Period Loss/Profit analysis with risk metrics and directional classification
+- **Market Regime Detection** - Identify bull/bear/sideways markets with volatility and strength metrics
+- **Seasonal Analysis** - Discover calendar-dependent patterns (monthly/quarterly performance)
+- **Sector Analysis** - Multi-stock pattern discovery with KBS blackboards per sector
+- **Walk-Forward Validation** - Time-series cross-validation to prevent pattern overfitting
+- **Pattern Context System** - Context-aware patterns that know when/where they're valid
 
-I am experimenting with different gems to support various functionality.  Sometimes they do not work out well.  For example I've gone through two different gems to implement the data frame capability.  Neither did what I wanted so I ended up creating my own data frame class based upon the old tried and true [Hashie](https://github.com/intridea/hashie) library.
+## Installation
 
+Install the gem:
 
-## Installation
+```bash
+gem install sqa
+```
 
-Install the gem and add to the application's Gemfile by executing:
+Or add to your `Gemfile`:
 
-    bundle add sqa
+```ruby
+gem 'sqa'
+```
 
-If bundler is not being used to manage dependencies, install the gem by executing:
+Then run:
 
-    gem install sqa
+```bash
+bundle install
+```
 
-## Semantic Versioning
+## Configuration
 
-`sqa` uses the [sem_version](https://github.com/canton7/sem_version) gem to provide a semantic version object.  Sure its old; but, so am I.  Doesn't mean we can't do the job.
+### Data Directory
+
+SQA stores historical price data and metadata in a data directory. By default, it uses `~/sqa_data/`:
 
 ```ruby
-# On the command line:
-sqa --version
+require 'sqa'
 
-# or inside your program:
+# Initialize with default configuration
+SQA.init
+
+# Or specify a custom data directory
+SQA::Config.new(data_dir: '~/Documents/my_stock_data')
+```
 
-SQA.version # returns SemVersion object
-exit(1) unless SQA.version >= SemVersion("1.0.0")
+Create your data directory:
 
-# Okay, you're right, you could put that kind of version
-# constraint in a Gemfile and let bundler handle the
-# dependencies for you.
+```bash
+mkdir ~/sqa_data
 ```
 
+### Alpha Vantage API Key
+
+SQA uses Alpha Vantage for stock data. You'll need a free API key from [https://www.alphavantage.co/](https://www.alphavantage.co/)
+
+Set the environment variable:
+
+```bash
+export AV_API_KEY="your_api_key_here"
+# or
+export ALPHAVANTAGE_API_KEY="your_api_key_here"
+```
+
+The free tier allows:
+- 5 API calls per minute
+- 100 API calls per day
+
 ## Usage
 
-**Do not use!** but its okay to play with.
+### Working with Stocks
 
-`SQA` can be used from the command line or as a library in your own application.
+```ruby
+require 'sqa'
+
+# Initialize SQA
+SQA.init
 
-`SQA` has a command line component.
+# Create or load a stock (automatically fetches/updates data from Alpha Vantage)
+aapl = SQA::Stock.new(ticker: 'AAPL', source: :alpha_vantage)
+#=> aapl with 1207 data points from 2019-01-02 to 2023-10-17
 
-```plaintext
-$ sqa --help
-Stock Quantitative Analysis (SQA)
+# Access the DataFrame
+aapl.df
+#=> SQA::DataFrame (wraps Polars::DataFrame)
 
-Usage: sqa [analysis|web] [OPTIONS]
+# Get column names
+aapl.df.columns
+#=> ["timestamp", "open_price", "high_price", "low_price", "close_price", "adj_close_price", "volume"]
 
-A collection of things
+# Access price data
+aapl.df["adj_close_price"]
+#=> Polars::Series with all adjusted closing prices
 
-Options:
-  -c, --config string             Path to the config file
-      --data-dir string           Set the directory for the SQA data
-  -d, --debug                     Turn on debugging output
-      --dump-config path_to_file  Dump the current configuration
-  -h, --help                      Print usage
-  -l, --log_level string          Set the log level (debug, info, warn, error,
-                                  fatal)
-  -p, --portfolio string          Set the filename of the portfolio
-  -t, --trades string             Set the filename into which trades are
-                                  stored
-  -v, --verbose                   Print verbosely
-      --version                   Print version
+# Get recent prices
+prices = aapl.df["adj_close_price"].to_a
+prices.last(5)
+#=> [179.8, 180.71, 178.85, 178.72, 177.15]
 
-Examples:
-  sqa -c ~/.sqa.yml -p portfolio.csv -t trades.csv --data-dir ~/sqa_data
+# Access company metadata
+aapl.name
+#=> "Apple Inc."
 
-  Optional Command Available:
-    analysis - Provide an Analysis of a Portfolio
-    web - Run a web server
+aapl.exchange
+#=> "NASDAQ"
 
-WARNING: This is a toy, a play thing, not intended for serious use.
+aapl.overview
+#=> Hash with detailed company information
 ```
 
-More about the two included programs `analysis` and `web` later.
+### DataFrame Operations
+
+SQA::DataFrame is a wrapper around the high-performance Polars DataFrame:
+
+```ruby
+require 'sqa'
+
+# Create from hash
+data = {
+  timestamp: ['2023-01-01', '2023-01-02', '2023-01-03'],
+  price: [100.0, 101.5, 99.8]
+}
+df = SQA::DataFrame.new(data)
+
+# Load from CSV
+df = SQA::DataFrame.load(source: 'aapl.csv')
+
+# Basic operations
+df.size        # Number of rows
+df.ncols       # Number of columns
+df.to_h        # Convert to hash
+df.to_csv('output.csv')  # Save to CSV
+
+# Access underlying Polars DataFrame
+df.data        # Polars::DataFrame
+```
 
-### Setup a Config File
+### Technical Indicators
 
-You will need to create a directory to store the `sqa` data and a configuration file.  You can start by doing this:
+All technical indicators are provided by the [`sqa-tai`](https://github.com/MadBomber/sqa-tai) gem, which wraps the industry-standard TA-Lib library:
 
 ```ruby
-gem install sqa
-mkdir ~/Documents/sqa_data
-sqa --data-dir ~/Documents/sqa_data --dump-config ~/.sqa.yml
+require 'sqa'
+
+prices = [100, 102, 105, 103, 107, 110, 108, 112, 115, 113]
+
+# Simple Moving Average
+sma = SQAI.sma(prices, period: 5)
+#=> [104.0, 105.4, 106.6, 108.0, 110.4, 111.6]
+
+# Relative Strength Index
+rsi = SQAI.rsi(prices, period: 14)
+#=> [70.5, 68.2, ...]
+
+# Exponential Moving Average
+ema = SQAI.ema(prices, period: 5)
+#=> [...]
+
+# Bollinger Bands (returns upper, middle, lower)
+upper, middle, lower = SQAI.bbands(prices, period: 5)
+
+# MACD
+macd, signal, histogram = SQAI.macd(prices, fast_period: 12, slow_period: 26, signal_period: 9)
+
+# Many more indicators available!
+# See: https://github.com/MadBomber/sqa-tai
 ```
 
-By default `SQA` looks for a configuration file named `.sqa.yml` in the current directory.  If it does not find one there it looks in the home directory.  You can use the `--config` CLI option to specify a path to your custom config file name.
+**Available Indicators:** SMA, EMA, WMA, RSI, MACD, Bollinger Bands, Stochastic, ADX, ATR, CCI, Williams %R, ROC, Momentum, and 140+ more via TA-Lib.
 
-You can have multiple configurations and multiple data directories.
+### Trading Strategies
 
-You can also have one data directory and multiple portfolio and trades files within that single directory.
+Build and test trading strategies:
 
-### AlphaVantage
+```ruby
+require 'sqa'
+require 'ostruct'
 
-`SQA` makes use of the `AlphaVantage` API to get some stock related information.  You will need an API key in order to use this functionality.  They have a free rate limited API key which allows 5 accesses per second; total of 100 accesses in a day.  If you are doing more than that you are not playing an ought to purchase one of there serious API key plans.
+# Load strategies
+strategies = SQA::Strategy.new
+strategies.auto_load  # Loads all built-in strategies
+
+# Add specific strategies
+strategies.add SQA::Strategy::RSI
+strategies.add SQA::Strategy::SMA
+strategies.add SQA::Strategy::EMA
+
+# Prepare data for strategy execution
+prices = aapl.df["adj_close_price"].to_a
+rsi_value = SQAI.rsi(prices, period: 14).last
+
+vector = OpenStruct.new
+vector.rsi = rsi_value
+vector.prices = prices
+
+# Execute strategies
+results = strategies.execute(vector)
+#=> [:hold, :buy, :hold]  # One result per strategy
+
+# Built-in strategies:
+# - SQA::Strategy::RSI - Based on Relative Strength Index
+# - SQA::Strategy::SMA - Simple Moving Average crossover
+# - SQA::Strategy::EMA - Exponential Moving Average crossover
+# - SQA::Strategy::MACD - MACD crossover strategy
+# - SQA::Strategy::BollingerBands - Bollinger Bands bounce strategy
+# - SQA::Strategy::Stochastic - Stochastic oscillator strategy
+# - SQA::Strategy::VolumeBreakout - Volume-based breakout strategy
+# - SQA::Strategy::MR - Mean Reversion
+# - SQA::Strategy::MP - Market Profile
+# - SQA::Strategy::KBS - Knowledge-based RETE strategy (advanced)
+# - SQA::Strategy::Random - Random signal generator (for testing)
+# - SQA::Strategy::Consensus - Combines multiple strategies
+```
 
-[https://www.alphavantage.co/](https://www.alphavantage.co/)
+### Statistics
 
-Put your API key in the system environment variable `AV_API_KEY`
+Comprehensive statistical analysis on price data (via `lite-statistics` gem):
 
-<!-- TODO: why is it not part of the configuration?  ought to be. -->
+```ruby
+require 'sqa'
 
+prices = [179.8, 180.71, 178.85, 178.72, 177.15]
+
+stats = prices.summary
+#=> {
+#     max: 180.71,
+#     min: 177.15,
+#     mean: 179.046,
+#     median: 178.85,
+#     mode: nil,
+#     range: 3.56,
+#     sample_standard_deviation: 1.19,
+#     sample_variance: 1.42,
+#     ...
+#   }
+```
 
-### TradingView
+### Portfolio Management
 
-**Not currently implemented** but seriously considering using [TradingView](http://tradingview.com) capabilities.  If I do choose to use these capabilities you will need your own API key or account or something.
+Track positions, trades, and P&L:
 
-Put your API key in the system environment variable `TV_API_KEY`
+```ruby
+require 'sqa'
 
-<!-- TODO: why is it not part of the configuration?  ought to be. -->
+# Create portfolio with initial cash and commission
+portfolio = SQA::Portfolio.new(initial_cash: 10_000.0, commission: 1.0)
 
-I kinda like their Terms of Use.  I've started crafting an [SQA Terms of Use](docs/terms_of_use.md) document based on theirs.  I specifically like sections 7 and 8.
+# Buy stock
+portfolio.buy('AAPL', shares: 10, price: 150.0, date: Date.today)
 
-### Yahoo Finance
+# Sell stock
+portfolio.sell('AAPL', shares: 5, price: 160.0, date: Date.today)
 
-The finance.yahoo.com website no longer supports an API.  To get information from this website you must scrape it.  That is how historical / receint stock prices are being obtained in the DataFrame::YahooFinance class.  I do not like doing this and do not necessarily recommend using YahooFinance as a source for stock price data.
+# Check portfolio value
+current_prices = { 'AAPL' => 165.0 }
+portfolio.value(current_prices)
+#=> Total portfolio value
 
-In the SQA::Stock class the default source is AlphaVantage which has its own limitations.
+# Calculate profit/loss
+portfolio.profit_loss(current_prices)
+#=> P&L amount
 
-This is not to say that finance.yahoo.com has no use.  In fact is is the perfect place to go to manually download CSV files of historical stock price data.  When you do that through your browser, the CSV file will be placed in your default downloads directory.
+# View summary
+portfolio.summary
+#=> { initial_cash: 10000.0, cash: 8248.0, positions: 1, trades: 2, ... }
 
-To use this downed file within the SQA environment it must be moved into your `SQA.`data_dir` with a filename that is all lowercase.  The filename must be the stock's symbol with a `.csv` extension.  For example if you downloaded the entire historical stock price data for Apple Computer (AAPL) the filename in the SQA.data_dir should be "aapl.csv"
+# Save trades to CSV
+portfolio.save_to_csv('my_trades.csv')
 
-You can manually go to the Yahoo Finance website at [https://finance.yahoo.com/quote/AAPL/history?p=AAPL](https://finance.yahoo.com/quote/AAPL/history?p=AAPL)
+# Load from CSV
+portfolio.load_from_csv('my_trades.csv')
+```
 
+### Backtesting
 
-## Playing in IRB
+Simulate trading strategies against historical data:
 
-You can play around in IRB with the SQA framework.
+```ruby
+require 'sqa'
 
+SQA.init
+stock = SQA::Stock.new(ticker: 'AAPL')
+
+# Create backtest
+backtest = SQA::Backtest.new(
+  stock: stock,
+  strategy: SQA::Strategy::RSI,
+  initial_capital: 10_000.0,
+  commission: 1.0
+)
+
+# Run backtest
+results = backtest.run
+
+# View comprehensive metrics
+puts "Total Return: #{results.total_return.round(2)}%"
+puts "Annualized Return: #{results.annualized_return.round(2)}%"
+puts "Sharpe Ratio: #{results.sharpe_ratio.round(2)}"
+puts "Max Drawdown: #{results.max_drawdown.round(2)}%"
+puts "Total Trades: #{results.total_trades}"
+puts "Win Rate: #{results.win_rate.round(2)}%"
+puts "Profit Factor: #{results.profit_factor.round(2)}"
+
+# Access equity curve for charting
+results.equity_curve  #=> Array of portfolio values over time
+```
 
-### With Stocks and Indicators
+### Strategy Generator
 
-You will need some CSV files.  If you ask for a stock to which you have not existing historical price data in a CSV file, `SQA` can use either Alpha Vantage or Yahoo Finance to get some data.  I like Alpha Vantage better because it has a well defined and documented API.  Yahoo Finance on the other hand does not.  You can manually download historical stock price data from Yahoo Finance into you `sqa data directory`
+Reverse-engineer profitable trades to discover winning patterns:
 
-Historical price data is kept in the `SQA.data_dir` in a CSV file whose name is all lowercase.  If you download the CSV file for the stock symbol "AAPL" it should be saved in you `SQA.data_dir` as `aapl.csv`
+```ruby
+require 'sqa'
 
-#### Get Historical Prices
+SQA.init
+stock = SQA::Stock.new(ticker: 'AAPL')
+
+# Create strategy generator
+generator = SQA::StrategyGenerator.new(
+  stock: stock,
+  min_gain_percent: 10.0,    # Find trades with ≥10% gain
+  holding_period: (5..20)    # Within 5-20 days
+)
+
+# Discover patterns
+patterns = generator.discover_patterns(min_pattern_frequency: 3)
+
+# Print discovered patterns
+generator.print_patterns(max_patterns: 10)
+#=> Pattern #1:
+#=>   Frequency: 15 occurrences
+#=>   Average Gain: 12.5%
+#=>   Average Holding: 8.3 days
+#=>   Conditions:
+#=>     - rsi: oversold
+#=>     - macd_crossover: bullish
+#=>     - volume: high
+
+# Generate strategy from top pattern
+strategy = generator.generate_strategy(pattern_index: 0, strategy_type: :class)
+
+# Backtest the discovered strategy
+backtest = SQA::Backtest.new(stock: stock, strategy: strategy)
+results = backtest.run
+
+# Export patterns to CSV
+generator.export_patterns('/tmp/patterns.csv')
+```
+
+### Genetic Programming
 
-Go to [https::/finance.yahoo.com](https::/finance.yahoo.com) and down some historical price data for your favorite stocks.  Put those CSV files into the `SQA.data_dir`.
+Evolve optimal strategy parameters through natural selection:
 
-You may need to create a `portfolio.csv` file or you may not.
+```ruby
+require 'sqa'
 
-<!-- TODO: Add a section on how to create a portfolio fCSV file -->
+SQA.init
+stock = SQA::Stock.new(ticker: 'AAPL')
+
+# Create genetic program
+gp = SQA::GeneticProgram.new(
+  stock: stock,
+  population_size: 50,
+  generations: 100,
+  mutation_rate: 0.15
+)
+
+# Define parameter space to explore
+gp.define_genes(
+  rsi_period: (7..30).to_a,
+  buy_threshold: (20..40).to_a,
+  sell_threshold: (60..80).to_a
+)
+
+# Define fitness function (backtest performance)
+gp.fitness do |genes|
+  # Create strategy with these genes
+  strategy = create_rsi_strategy(genes)
+
+  # Backtest and return total return as fitness
+  backtest = SQA::Backtest.new(stock: stock, strategy: strategy)
+  backtest.run.total_return
+end
+
+# Evolve!
+best = gp.evolve
+
+puts "Best Parameters:"
+puts "  RSI Period: #{best.genes[:rsi_period]}"
+puts "  Buy Threshold: #{best.genes[:buy_threshold]}"
+puts "  Sell Threshold: #{best.genes[:sell_threshold]}"
+puts "  Fitness: #{best.fitness.round(2)}%"
+
+# View evolution history
+gp.history.each do |gen|
+  puts "Gen #{gen[:generation]}: Best=#{gen[:best_fitness].round(2)}%"
+end
+```
 
-The CSV files will be named by the stock's ticker symbol as lower case.  For example: `aapl.csv`
+### Knowledge-Based Strategy (KBS)
 
-### Playing in the IRB - Setup
+Build sophisticated rule-based systems with RETE forward chaining:
 
 ```ruby
 require 'sqa'
-require 'sqa/cli'
 
-# You can pass a set of CLI options in a String
-SQA.init "-c ~/.sqa.yml"
+# Create KBS strategy
+strategy = SQA::Strategy::KBS.new(load_defaults: false)
+
+# Define custom trading rules
+strategy.add_rule :golden_opportunity do
+  desc "Perfect storm: Multiple bullish indicators align"
+
+  # Multiple conditions
+  on :rsi, { level: :oversold }
+  on :macd, { crossover: :bullish }
+  on :stochastic, { zone: :oversold, crossover: :bullish }
+  on :trend, { short_term: :up, strength: :strong }
+  on :volume, { level: :high }
+
+  # Negation: Don't buy if overbought elsewhere
+  without :rsi, { level: :overbought }
+
+  # Action
+  then do
+    assert(:signal, { action: :buy, confidence: :high })
+  end
+end
+
+# Execute strategy
+signal = strategy.execute(vector)
+#=> :buy or :sell or :hold
+
+# Use with backtesting
+backtest = SQA::Backtest.new(
+  stock: stock,
+  strategy: SQA::Strategy::KBS,
+  initial_capital: 10_000.0
+)
+results = backtest.run
+```
+
+### Real-Time Streaming
 
-# If you have an API key for AlphaVantage you can create a new
-# CSV file, or update an existing one.  Your key MUST be
-# in the system environment variable AV_API_KEY
-aapl = SQA::Stock.new(ticker: 'aapl', source: :alpha_vantage)
-#=> aapl with 1207 data points from 2019-01-02 to 2023-10-17
+Process live stock prices and generate on-the-fly trading signals:
+
+```ruby
+require 'sqa'
+
+# Create stream processor
+stream = SQA::Stream.new(
+  ticker: 'AAPL',
+  window_size: 100,
+  strategies: [
+    SQA::Strategy::RSI,
+    SQA::Strategy::MACD,
+    SQA::Strategy::BollingerBands
+  ]
+)
+
+# Register signal callback
+stream.on_signal do |signal, data|
+  puts "🔔 SIGNAL: #{signal.upcase}"
+  puts "   Price: $#{data[:price]}"
+  puts "   Consensus: #{data[:strategies_vote]}"
+
+  # Execute trade, send alert, etc.
+  execute_trade(signal, data) if signal != :hold
+end
+
+# Register update callback (optional)
+stream.on_update do |data|
+  puts "📊 Price update: $#{data[:price]}"
+end
+
+# Feed live data (from WebSocket, API, etc.)
+stream.update(
+  price: 150.25,
+  volume: 1_000_000,
+  timestamp: Time.now
+)
+
+# Access real-time indicators
+rsi = stream.indicator(:rsi, period: 14)
+sma = stream.indicator(:sma, period: 20)
+
+# View stream statistics
+stream.stats
+#=> { ticker: 'AAPL', updates: 125, current_price: 150.25, ... }
 ```
 
-`aapl.df`   is the data frame.  It is implemented as a Hashie::Mash obect -- a Hash or Arrays.
+### FPL Analysis (Future Period Loss/Profit)
+
+Analyze potential future price movements with risk metrics:
 
 ```ruby
-aapl.df.keys
-#=> [:timestamp, :open_price, :high_price, :low_price, :close_price, :adj_close_price, :volume]
+require 'sqa'
 
-aapl.df.adj_close_price.last(5)
-#=> [179.8, 180.71, 178.85, 178.72, 177.15]
+stock = SQA::Stock.new(ticker: 'AAPL')
+
+# Calculate FPL for 14-day future period
+fpl_data = stock.df.fpl(fpop: 14)
+# Returns: [[min_delta, max_delta], ...] for each price point
+
+# Full analysis with risk metrics
+analysis = stock.df.fpl_analysis(fpop: 14)
+
+analysis.first
+#=> {
+#  min_delta: -5.2,      # Worst loss percentage
+#  max_delta: 8.7,       # Best gain percentage
+#  risk: 13.9,           # Volatility (max - min)
+#  direction: :UNCERTAIN,# :UP, :DOWN, :UNCERTAIN, :FLAT
+#  magnitude: 1.75,      # Average expected movement
+#  interpretation: "UNCERTAIN: 1.75% (±6.95% risk)"
+#}
+
+# Filter high-quality opportunities
+good_indices = SQA::FPOP.filter_by_quality(
+  analysis,
+  min_magnitude: 5.0,     # Minimum average movement
+  max_risk: 10.0,         # Maximum acceptable volatility
+  directions: [:UP]       # Only bullish patterns
+)
+
+# Use with Strategy Generator for risk-aware pattern discovery
+generator = SQA::StrategyGenerator.new(
+  stock: stock,
+  min_gain_percent: 10.0,
+  fpop: 10,
+  max_fpl_risk: 20.0,           # NEW: Filter high-volatility points
+  required_fpl_directions: [:UP] # NEW: Only bullish patterns
+)
 ```
 
-`aapl.data` is basic static data, company name, industry etc.  It is also implemented as a Hassie::Mash object but is primary treated as a plain hash object.
+**Key Benefits:**
+- Captures both upside AND downside potential
+- Measures volatility/risk during future period
+- Classifies directional bias (UP/DOWN/UNCERTAIN/FLAT)
+- Enables risk-adjusted opportunity filtering
+
+### Market Regime Detection
+
+Identify and adapt to changing market conditions:
 
 ```ruby
-aapl.data.keys
-#=> [:ticker, :source, :indicators, :overview]
+# Detect current market regime
+regime = SQA::MarketRegime.detect(stock)
+#=> {
+#  type: :bull,           # :bull, :bear, :sideways
+#  volatility: :low,      # :low, :medium, :high
+#  strength: :strong,     # :weak, :moderate, :strong
+#  lookback_days: 60,
+#  detected_at: 2024-11-08
+#}
+
+# Analyze regime history
+regimes = SQA::MarketRegime.detect_history(stock)
+#=> [
+#  { type: :bull, start_index: 0, end_index: 120, duration: 120 },
+#  { type: :bear, start_index: 121, end_index: 200, duration: 79 },
+#  ...
+#]
+
+# Split data by regime
+regime_data = SQA::MarketRegime.split_by_regime(stock)
+#=> {
+#  bull: [{ prices: [...], start_index: 0, end_index: 120 }, ...],
+#  bear: [...],
+#  sideways: [...]
+#}
+
+# Use regime-specific strategies
+current_regime = SQA::MarketRegime.detect(stock)
+strategy = case current_regime[:type]
+when :bull
+  SQA::Strategy::MomentumBreakout
+when :bear
+  SQA::Strategy::MeanReversion
+when :sideways
+  SQA::Strategy::BollingerBands
+end
+```
+
+**Key Benefits:**
+- Different strategies for different market conditions
+- Avoid using bull market strategies in bear markets
+- Detect regime changes before they impact performance
 
-aapl.data.source
-=> "alpha_vantage"
+### Seasonal Analysis
 
-aapl.data.overciew.keys
-=> [:symbol, :asset_type, :name, :description, :cik, :exchange, :currency, :country, :sector, :industry, :address, :fiscal_year_end, :latest_quarter, :market_capitalization, :ebitda, :pe_ratio, :peg_ratio, :book_value, :dividend_per_share, :dividend_yield, :eps, :revenue_per_share_ttm, :profit_margin, :operating_margin_ttm, :return_on_assets_ttm, :return_on_equity_ttm, :revenue_ttm, :gross_profit_ttm, :diluted_epsttm, :quarterly_earnings_growth_yoy, :quarterly_revenue_growth_yoy, :analyst_target_price, :trailing_pe, :forward_pe, :price_to_sales_ratio_ttm, :price_to_book_ratio, :ev_to_revenue, :ev_to_ebitda, :beta, :"52_week_high", :"52_week_low", :"50_day_moving_average", :"200_day_moving_average", :shares_outstanding, :dividend_date, :ex_dividend_date]
+Discover calendar-dependent patterns:
 
+```ruby
+# Analyze seasonal performance
+seasonal = SQA::SeasonalAnalyzer.analyze(stock)
+#=> {
+#  best_months: [10, 11, 12],        # October, November, December
+#  worst_months: [5, 6, 7],          # May, June, July
+#  best_quarters: [4, 1],            # Q4 and Q1
+#  worst_quarters: [2, 3],
+#  has_seasonal_pattern: true,
+#  monthly_returns: { ... },
+#  quarterly_returns: { ... }
+#}
+
+# Filter data for Q4 only (holiday shopping season)
+q4_data = SQA::SeasonalAnalyzer.filter_by_quarters(stock, [4])
+#=> { indices: [...], dates: [...], prices: [...] }
+
+# Filter for specific months
+dec_data = SQA::SeasonalAnalyzer.filter_by_months(stock, [12])
+
+# Get context for specific date
+context = SQA::SeasonalAnalyzer.context_for_date(Date.new(2024, 12, 15))
+#=> {
+#  month: 12,
+#  quarter: 4,
+#  month_name: "December",
+#  quarter_name: "Q4",
+#  is_year_end: true,
+#  is_holiday_season: true,
+#  is_earnings_season: false
+#}
+
+# Discover patterns only for best months
+generator = SQA::StrategyGenerator.new(stock: stock)
+patterns = generator.discover_context_aware_patterns(
+  analyze_seasonal: true
+)
+
+patterns.each do |pattern|
+  puts pattern.context.valid_months  # [10, 11, 12] for Q4 patterns
+end
 ```
 
-### Playing in the IRB - Statistics
+**Real-World Examples:**
+- Retail stocks: Q4 holiday shopping boost
+- Tax prep stocks: Q1/Q4 seasonal surge
+- Energy stocks: Summer driving season
+- Agriculture stocks: Planting/harvest cycles
+
+### Sector Analysis with KBS
 
-Basic statistics are available on all of the SQA::DataFrame arrays.
+Analyze patterns across related stocks using KBS blackboards:
 
 ```ruby
-require 'amazing_print' # to get the ap command
-
-# Look at some summary stats on the last 5 days of
-# adjusted closing pricess of AAPL
-ap aapl.df.adj_close_price.last(5).summary
-{
-                        :frequencies => {
-         179.8 => 1,
-        180.71 => 1,
-        178.85 => 1,
-        178.72 => 1,
-        177.15 => 1
-    },
-                                :max => 180.71,
-                               :mean => 179.046,
-                             :median => 178.85,
-                           :midrange => 178.93,
-                                :min => 177.15,
-                               :mode => nil,
-                        :proportions => {
-         179.8 => 0.2,
-        180.71 => 0.2,
-        178.85 => 0.2,
-        178.72 => 0.2,
-        177.15 => 0.2
-    },
-                          :quartile1 => 178.85,
-                          :quartile2 => 179.8,
-                          :quartile3 => 180.71,
-                              :range => 3.5600000000000023,
-                               :size => 5,
-                                :sum => 895.23,
-    :sample_coefficient_of_variation => 0.006644656242680533,
-                    :sample_kurtosis => 2.089087404921432,
-                        :sample_size => 5,
-                    :sample_skewness => -0.2163861377512453,
-          :sample_standard_deviation => 1.1896991216269788,
-              :sample_standard_error => 0.532049621745943,
-                    :sample_variance => 1.415384000000005,
-                     :sample_zscores => {
-         179.8 => 0.6337736880639895,
-        180.71 => 1.3986729667618856,
-        178.85 => -0.16474753695031497,
-        178.72 => -0.2740188624785824,
-        177.15 => -1.5936802553969298
-    }
-}
-#=> nil
+# Create sector analyzer (separate blackboard per sector)
+analyzer = SQA::SectorAnalyzer.new
+
+# Add stocks to technology sector
+analyzer.add_stock('AAPL', sector: :technology)
+analyzer.add_stock('MSFT', sector: :technology)
+analyzer.add_stock('GOOGL', sector: :technology)
+
+# Detect sector-wide regime
+tech_stocks = [
+  SQA::Stock.new(ticker: 'AAPL'),
+  SQA::Stock.new(ticker: 'MSFT'),
+  SQA::Stock.new(ticker: 'GOOGL')
+]
+
+sector_regime = analyzer.detect_sector_regime(:technology, tech_stocks)
+#=> {
+#  sector: :technology,
+#  consensus_regime: :bull,  # Majority vote
+#  sector_strength: 85.0,    # % of stocks bullish
+#  stock_regimes: [...]
+#}
+
+# Discover patterns across entire sector
+sector_patterns = analyzer.discover_sector_patterns(
+  :technology,
+  tech_stocks,
+  min_gain_percent: 10.0,
+  fpop: 10
+)
+
+# Patterns that work for MULTIPLE stocks in sector
+sector_patterns.each do |sp|
+  puts "Pattern found in: #{sp[:stocks].join(', ')}"
+  puts "Avg gain: #{sp[:avg_gain].round(2)}%"
+  puts "Conditions: #{sp[:conditions]}"
+end
+
+# Query sector blackboard
+facts = analyzer.query_sector(:technology, :sector_pattern)
+
+# Print sector summary
+analyzer.print_sector_summary(:technology)
 ```
 
-### Playing in the IRB - Indicators
+**Available Sectors:**
+`:technology`, `:finance`, `:healthcare`, `:energy`, `:consumer`, `:industrial`, `:materials`, `:utilities`, `:real_estate`, `:communications`
 
-The SQA::Indicator class methods use Arrays not the DataFrame
-Here is an example:
+**Key Benefits:**
+- Leverage "stocks move together" in same sector
+- Find patterns valid across multiple related stocks
+- Persistent KBS blackboard tracks sector knowledge
+- Cross-stock pattern validation
 
+### Walk-Forward Validation
+
+Prevent overfitting with time-series cross-validation:
 
 ```ruby
-prices = aapl.df.adj_close_price
-period = 14 # size of the window in prices to analyze
+generator = SQA::StrategyGenerator.new(
+  stock: stock,
+  min_gain_percent: 10.0,
+  fpop: 10
+)
+
+# Run walk-forward validation
+results = generator.walk_forward_validate(
+  train_size: 250,  # 1 year training window
+  test_size: 60,    # 3 months testing window
+  step_size: 30     # Step forward 1 month each iteration
+)
+
+# Only patterns that work out-of-sample
+validated_patterns = results[:validated_patterns]
+
+# Detailed results for each iteration
+results[:validation_results].each do |r|
+  puts "Train: #{r[:train_period]}"
+  puts "Test:  #{r[:test_period]}"
+  puts "Return: #{r[:test_return]}%"
+  puts "Sharpe: #{r[:test_sharpe]}"
+end
+
+# Statistics
+all_returns = results[:validation_results].map { |r| r[:test_return] }
+avg_return = all_returns.sum / all_returns.size
+puts "Average out-of-sample return: #{avg_return.round(2)}%"
+```
+
+**How It Works:**
+1. Split data into train/test windows
+2. Discover patterns on training data
+3. Test patterns on future (unseen) data
+4. Roll forward and repeat
+5. Only keep patterns that work out-of-sample
+
+**Prevents:**
+- Curve-fitting to historical noise
+- Patterns that only worked in the past
+- Overconfidence in backtest results
+
+### Pattern Context System
 
-rsi = SQA::Indicator.rsi(prices, period)
-#=> {:rsi=>63.46652828230407, :trend=>:normal}
+Context-aware patterns that know when/where they're valid:
+
+```ruby
+# Discover patterns with full context
+generator = SQA::StrategyGenerator.new(stock: stock, min_gain_percent: 10.0)
+
+patterns = generator.discover_context_aware_patterns(
+  analyze_regime: true,
+  analyze_seasonal: true,
+  sector: :technology
+)
+
+# Each pattern has rich context metadata
+pattern = patterns.first
+pattern.context.market_regime        #=> :bull
+pattern.context.volatility_regime    #=> :low
+pattern.context.valid_months         #=> [10, 11, 12]
+pattern.context.valid_quarters       #=> [4, 1]
+pattern.context.sector               #=> :technology
+pattern.context.discovered_period    #=> "2020-01-01 to 2022-12-31"
+
+# Runtime validation - check if pattern applies NOW
+valid = pattern.context.valid_for?(
+  date: Date.today,
+  regime: :bull,
+  sector: :technology
+)
+
+if valid
+  # Execute strategy - context matches current conditions
+  signal = strategy.trade(vector)
+else
+  # Skip - pattern not valid in current context
+  puts "Pattern not valid for current conditions"
+end
+
+# Pattern summary includes context
+puts pattern.to_s
+#=> "Pattern(conditions=3, freq=12, gain=11.5%, success=75.0%) [bull months:10,11,12 technology]"
 ```
 
-### Playing in the IRB - Strategies
+**Context Metadata:**
+- **Market regime**: Bull/bear/sideways classification
+- **Seasonality**: Valid months/quarters
+- **Sector**: Industry classification
+- **Volatility regime**: Low/medium/high
+- **Discovery period**: When pattern was found
+- **Stability score**: Consistency over time
+
+**Key Benefits:**
+- Patterns aren't universal - they're context-specific
+- Avoid using patterns in wrong market conditions
+- Seasonal awareness (Q4 retail patterns, etc.)
+- Sector-specific strategies
+- Runtime validation before execution
+
+## Interactive Console
+
+Launch an interactive Ruby console with SQA loaded:
+
+```bash
+sqa-console
+```
 
-The strategies work off of an Object that contains the information required to make its recommendation.  Building on the previous Ruby snippet ...
+This opens IRB with the SQA library pre-loaded, allowing you to experiment interactively:
 
 ```ruby
-require 'ostruct'
-vector     = OpenStruct.new
-vector.rsi = rsi
-
-# Load some trading strategies
-ss = SWA::Strategy.new
-ss.auto_load # loads everything in the lib/sqa/strategy folder
-
-# Select some strategies to execute
-ss.add SWA::Strategy::Random # 3-sided coin flip
-ss.add SQA::Strategy::RSI
-
-# This is an Array with each "trade" method
-# that is defined in each strategy added
-ap ss.strategies #=>∑
-[
-    [0] SQA::Strategy::Random#trade(vector),
-    [1] SQA::Strategy::RSI#trade(vector)
-]
+# Already loaded: require 'sqa'
+
+SQA.init
+stock = SQA::Stock.new(ticker: 'MSFT')
+prices = stock.df["close_price"].to_a
+sma = SQAI.sma(prices, period: 20)
+```
+
+## Examples
+
+The `examples/` directory contains comprehensive demonstrations of advanced features:
+
+- **`genetic_programming_example.rb`** - Evolve RSI parameters through natural selection
+- **`kbs_strategy_example.rb`** - Build rule-based trading systems with RETE
+- **`realtime_stream_example.rb`** - Process live price streams with callbacks
+- **`strategy_generator_example.rb`** - Mine profitable patterns from history
 
-# Execute those strategies
-results = ss.execute(vector)
-#=> [:hold, :hold]
+Run examples:
+```bash
+ruby examples/genetic_programming_example.rb
+ruby examples/strategy_generator_example.rb
 ```
 
-`results` is an Array with an entry for each strategy executed.  The entries are either :buy, :sell or :hold.
+See `examples/README.md` for detailed documentation and integration patterns.
 
-Currently the strategies are executed sequentially so the results can easily be mapped back to which strategy produced which result.  In the future that will change so that the strategies are executed concurrently.  When that change is introduced the entries in the `results` object will change -- most likely to an Array of Hashes.
+## Architecture
 
-Any specific strategy may not work on every stock. Using the historical data, it is possible to see which strategy works better for a specific stock.  **Of course the statistical motto is that historical performance is not a fail-proof indicator for future performance.**
+**Core Components:**
 
-The strategies that come with the `SQA::Strategy` class are examples only.  Its expected that you will come up with your own.  If you do, consider sharing them.
+- **`SQA::DataFrame`** - High-performance Polars-based data container for time series
+- **`SQA::Stock`** - Represents a stock with price history and metadata
+- **`SQA::Ticker`** - Stock symbol validation and lookup
+- **`SQA::Strategy`** - Trading strategy execution framework
+- **`SQA::Config`** - Configuration management
+- **`SQAI`** - Alias for `SQA::TAI` (technical indicators from sqa-tai gem)
 
-## Included Program Examples
+**Advanced Components:**
 
-<!-- TODO: What is the name of these things?  From the help text they are called commands.  They are treated in the code base as a class under the SQA module.  Its easy to change the CLI help text to call these programs rather than commands. -->
+- **`SQA::Portfolio`** - Position and trade tracking with P&L calculations
+- **`SQA::Backtest`** - Strategy simulation with comprehensive metrics
+- **`SQA::StrategyGenerator`** - Pattern mining from profitable historical trades
+- **`SQA::GeneticProgram`** - Evolutionary algorithm for parameter optimization
+- **`SQA::Strategy::KBS`** - RETE-based forward-chaining inference engine
+- **`SQA::Stream`** - Real-time price stream processor with event callbacks
 
-There are at least two included programs that make use of the `SQA` library.
+**Data Flow:**
 
-<!-- TODO: These kinds of things need to be in their own repository. -->
+1. Create `SQA::Stock` with ticker symbol
+2. Stock fetches data from Alpha Vantage or Yahoo Finance
+3. Data stored in Polars-based `SQA::DataFrame`
+4. Apply technical indicators via `SQAI` / `SQA::TAI`
+5. Execute trading strategies to generate signals
+6. Analyze results with statistical functions
 
-<!-- TODO: Need to complete the API for invoking one of these "programs" from the sqa cli. -->
+**Design Patterns:**
 
-### Analysis
+- Plugin architecture for indicators and strategies
+- Data source abstraction (Alpha Vantage, Yahoo Finance)
+- Delegation to Polars for DataFrame operations
+- Configuration hierarchy: defaults < environment variables < config file
 
-Does an analysis of a portfollio of stocks.
+## Data Sources
 
-### Web
+### Alpha Vantage
 
-Provides a browser-based interface to some stuff.
+**Recommended data source** with a well-documented API.
+
+- **URL:** [https://www.alphavantage.co/](https://www.alphavantage.co/)
+- **API Key:** Required (free tier available)
+- **Environment Variable:** `AV_API_KEY` or `ALPHAVANTAGE_API_KEY`
+- **Rate Limits:** 5 calls/minute, 100 calls/day (free tier)
+
+```ruby
+stock = SQA::Stock.new(ticker: 'GOOGL', source: :alpha_vantage)
+```
+
+### Yahoo Finance
+
+**No API available** - uses web scraping for historical data.
+
+- **URL:** [https://finance.yahoo.com/](https://finance.yahoo.com/)
+- **Manual Download:** Download CSV files and place in `SQA.data_dir`
+- **Filename Format:** `ticker.csv` (lowercase), e.g., `aapl.csv`
+
+To manually download:
+1. Visit [https://finance.yahoo.com/quote/AAPL/history?p=AAPL](https://finance.yahoo.com/quote/AAPL/history?p=AAPL)
+2. Download historical data as CSV
+3. Move to your data directory as `aapl.csv`
+
+```ruby
+stock = SQA::Stock.new(ticker: 'AAPL', source: :yahoo_finance)
+```
 
 ## Contributing
 
-I can always use some help on this stuff.  Got an idea for a new indicator or strategy?  Want to improve the math?  Make the signals better?  Let's collaborate!
+Contributions are welcome! Got an idea for a new indicator or strategy? Want to improve the math or signals?
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/MadBomber/sqa.
+- **Bug reports and pull requests:** [https://github.com/MadBomber/sqa](https://github.com/MadBomber/sqa)
+- **Technical indicators:** Contribute to [sqa-tai](https://github.com/MadBomber/sqa-tai)
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+---
+
+**Remember:** This is an educational tool. Historical performance is not an indicator of future results. Never use this for real trading decisions.