sqa-bi 0.1.0

data/docs/getting-started/core-concepts.md

@@ -0,0 +1,119 @@
+# Core Concepts
+
+Four ideas carry the whole library: **outcomes**, **features**, **Bayes'
+theorem**, and **the posterior as the deliverable**.
+
+## Outcomes are discrete and ordered
+
+`SQA::BI` predicts over a finite set you choose up front:
+
+```ruby
+outcomes: [-2, -1, 0, 1, 2]
+```
+
+Two properties matter.
+
+**Discrete.** Not a continuous value. You are not predicting "+2.7%", you are
+predicting which *band* the move falls in. That is a deliberate reduction: bands
+collect enough observations per class for density estimation to work, and they
+map directly onto decisions ("act" / "wait" / "act the other way").
+
+**Ordered.** `-2 < -1 < 0 < 1 < 2` means something. Being wrong by one level is
+a smaller error than being wrong by four. The library does not *enforce* the
+ordering, but the outcome set is sorted on construction and every report prints
+in order, so the structure stays visible.
+
+??? tip "How many levels?"
+    Three is the minimum that says anything ("down / flat / up"). Five is the
+    common default. Beyond about seven, each level starts starving for
+    observations and the KDE estimate gets noisy — you gain resolution on paper
+    and lose it in practice. If you need more granularity, get more data first.
+
+## Features are a numeric vector
+
+Every observation pairs a feature vector with the outcome that followed:
+
+```ruby
+{ features: [1.0, 2.0, 3.0], outcome: 1 }
+```
+
+The vector must be numeric and every vector must have the same length. Beyond
+that the library is indifferent to what the numbers mean.
+
+!!! important "Prefer deltas over levels"
+    Feed *changes*, not absolute values. A close price of 187.40 tells the model
+    nothing transferable; a close price change of +0.8% does. Absolute levels
+    make the KDE memorize a price regime it will never see again — the stock
+    demo therefore builds every feature as a delta from the previous day.
+
+    The same logic applies outside markets: rate of change generalizes,
+    raw level rarely does.
+
+## Bayes' theorem, in three objects
+
+![Prior × Likelihood → Posterior](../assets/diagrams/bayes-pipeline.svg)
+
+$$
+P(\text{outcome} \mid \text{data}) =
+\frac{P(\text{data} \mid \text{outcome}) \times P(\text{outcome})}{P(\text{data})}
+$$
+
+Each term is a class:
+
+**[`Prior`](../guide/prior.md) — `P(outcome)`**
+: What you believed before seeing this feature vector. Uniform by default.
+  Can be set explicitly, learned from observed frequencies with Laplace
+  smoothing, or [elicited from an LLM](../llm/prior-elicitation.md).
+
+**[`Likelihood`](../guide/likelihood.md) — `P(data | outcome)`**
+: How typical these features are of each outcome, estimated by Gaussian kernel
+  density over the observations you trained on. This is where the data enters.
+
+**[`Posterior`](../guide/posterior.md) — `P(outcome | data)`**
+: The product, normalized. Your updated belief.
+
+`P(data)`, the evidence, is just the normalizing constant — the sum over all
+outcomes. Ruby computes it exactly, every time.
+
+## The posterior is the deliverable
+
+Most classifiers hand back a label and discard everything else. Here the
+distribution *is* the result, and it carries its own quality report:
+
+| Measure | Range | Reading |
+|---|---|---|
+| `entropy` | 0 → log₂(n) | 0 = certain; log₂(5) ≈ 2.322 = a coin toss over 5 outcomes |
+| `confidence` | 0 → 1 | `1 − entropy / max_entropy`; 0 = uniform, 1 = certain |
+| `kl_divergence_from_prior` | ≥ 0 | bits of information this evidence added |
+
+The third one answers a question a label never can: *did this observation
+actually teach me anything?* A KL divergence near zero means the features moved
+your belief hardly at all — the prediction is just your prior in a costume.
+
+See [Quantifying Uncertainty](../guide/uncertainty.md) for how to use each.
+
+## How the pieces connect
+
+![SQA::BI architecture](../assets/diagrams/architecture.svg)
+
+[`TimeSeriesPredictor`](../guide/predictor.md) is the facade. It owns the
+outcome set and bandwidth, holds a `Prior` and a `Likelihood`, and builds a
+fresh `Posterior` on every `predict`. You can use the three underlying classes
+directly when you want finer control — they have no hidden coupling.
+
+## What this is not
+
+**Not a time series model.** Despite the name, `TimeSeriesPredictor` has no
+notion of sequence, autocorrelation, or seasonality. It maps a feature vector to
+a distribution. Any temporal structure you want must be *encoded into the
+features* — a lag, a rolling mean, a momentum delta.
+
+**Not multi-horizon.** One predictor answers one question at one horizon. For
+k = 1…10 days you currently need ten predictors, and they will not share
+strength across horizons even though adjacent horizons are strongly correlated.
+That limitation is [a known open thread](../EXPLORATION.md).
+
+**Not calibrated by construction.** A Bayesian posterior is coherent, which is
+not the same as calibrated. If your priors are wrong or your features are
+uninformative, the numbers will be confidently wrong in a mathematically
+impeccable way. Backtest before believing.

data/docs/getting-started/installation.md

@@ -0,0 +1,112 @@
+# Installation
+
+## Requirements
+
+- **Ruby >= 3.2.0** — the workspace's active development Ruby is 4.0.x via rbenv
+- **No runtime dependencies.** The core math is plain Ruby.
+
+`ruby_llm` is needed only for the LLM-backed prior and likelihood paths, and it
+is required *lazily* inside `LlmSupport.build_chat`. The library loads, and
+kernel density estimation works, without it installed.
+
+## As a gem
+
+```bash
+gem install sqa-bi
+```
+
+Or in a `Gemfile`:
+
+```ruby
+gem "sqa-bi"
+```
+
+Then:
+
+```ruby
+require "sqa/bi"
+```
+
+!!! note "The require path is `sqa/bi`, not `sqa-bi`"
+    The gem is named `sqa-bi` with a hyphen, matching `sqa-tai`, but the entry
+    point is `lib/sqa/bi.rb`. `require "sqa-bi"` raises `LoadError`.
+
+## Optional — the LLM paths
+
+To use [`LlmPriorElicitor`](../llm/prior-elicitation.md) or
+[`LlmLikelihoodEstimator`](../llm/likelihood-estimation.md):
+
+```ruby
+gem "ruby_llm", ">= 2.0"
+
+# Local providers, required lazily. Their absence disables only the
+# corresponding provider; it never breaks the cloud path or the KDE path.
+gem "ruby_llm-providers-lms",   require: false  # LM Studio
+gem "ruby_llm-providers-apfel", require: false  # Apple Foundation Models
+```
+
+See [Providers and Local Models](../llm/providers.md) for the setup that makes
+local inference actually engage — which takes more than installing these gems.
+
+## Development in the SQA workspace
+
+`sqa-bi` lives in the `sqa_project` workspace alongside `sqa`, `sqa-tai`,
+`sqa-cli`, `sqa-advisor`, and the two demo apps.
+
+```bash
+cd ~/sandbox/git_repos/madbomber/projects/sqa_project/sqa-bi
+bundle install
+bundle exec rake test
+```
+
+### Bundle mode
+
+The workspace root `.envrc` reads `.bundle_mode` and exports `BUNDLE_GEMFILE`
+for every repo:
+
+| Mode | Gemfile | Resolves |
+|---|---|---|
+| `dev` (default) | `Gemfile.local` | sibling checkouts via `path:` overrides |
+| `prod` | `Gemfile` | released gems only |
+
+Switch from the workspace root with `asgard dev` / `asgard prod`; direnv reloads
+every shell at its next prompt.
+
+`sqa-bi`'s `Gemfile.local` adds `sqa` and `sqa-tai` from sibling checkouts. It
+does that **only** so the market-facing demo can run — the gem itself must never
+gain a `sqa` dependency, or the dependency graph inverts.
+
+??? info "The `ruby_llm` version conflict, and why it resolves"
+    `sqa` depends on `shared_tools`, which depends on `ruby_llm-mcp`, which pins
+    `ruby_llm ~> 1.9`. Meanwhile `sqa` itself and both local provider gems
+    require `ruby_llm >= 2.0`. That looks unresolvable.
+
+    Bundler handles it by backing `shared_tools` down to 0.2.3, which has no
+    `ruby_llm-mcp` dependency at all — the same resolution `sqa`'s own
+    `Gemfile.local.lock` lands on. Verified: `Gemfile.local` here resolves
+    `sqa 0.4.0` + `ruby_llm 2.0.0` + `shared_tools 0.2.3`, and the stock demo
+    runs end to end under `bundle exec`.
+
+    If a future `shared_tools` release drops the pin, this note becomes moot.
+
+### Quality gates
+
+```bash
+asgard test        # or: bundle exec rake test
+asgard quality     # tests + coverage, Flog, Flay, Reek
+asgard rubocop
+```
+
+See [Development](../development.md) for what each gate enforces and which
+smells are deliberately grandfathered.
+
+## Documentation
+
+This site is built with MkDocs and the Material theme:
+
+```bash
+mkdocs serve        # live reload at http://127.0.0.1:8000
+mkdocs build        # static site into site/
+```
+
+`site/` is gitignored.

data/docs/getting-started/quick-start.md

@@ -0,0 +1,143 @@
+# Quick Start
+
+A working predictor, end to end, in one file.
+
+## 1. Decide what you are predicting
+
+Outcomes are a small, **ordered** set. Five levels is a good default — enough
+resolution to be useful, few enough that each level collects observations:
+
+```ruby
+require "sqa/bi"
+
+OUTCOMES = [-2, -1, 0, 1, 2]
+LABELS   = {
+  -2 => "strong downtrend",
+  -1 => "mild downtrend",
+   0 => "sideways",
+   1 => "mild uptrend",
+   2 => "strong uptrend"
+}
+```
+
+The integers are just keys — any comparable object works — but signed integers
+centered on zero make the ordering obvious and read well in output.
+
+## 2. Build a predictor
+
+```ruby
+predictor = SQA::BI.predictor(outcomes: OUTCOMES, bandwidth: 1.0)
+```
+
+`bandwidth` controls how sharply the likelihood distinguishes nearby feature
+vectors. Start at `1.0` and tune once you have data — see
+[Tuning](../guide/tuning.md).
+
+## 3. Train on history
+
+Every observation is a numeric feature vector plus the outcome that followed it.
+
+```ruby
+predictor.train([0.8, 1.9, 3.1], outcome:  1)
+predictor.train([1.0, 2.0, 3.0], outcome:  1)
+predictor.train([1.1, 2.1, 2.9], outcome:  1)
+predictor.train([0.2, 0.1, 0.0], outcome:  0)
+predictor.train([-1.0, -2.0, 0.5], outcome: -2)
+```
+
+Or in bulk:
+
+```ruby
+observations = [
+  { features: [0.8, 1.9, 3.1], outcome:  1 },
+  { features: [1.0, 2.0, 3.0], outcome:  1 },
+  { features: [-1.0, -2.0, 0.5], outcome: -2 }
+]
+predictor.train_batch(observations)
+```
+
+!!! warning "Feature vectors must all have the same dimension"
+    The first `train` call fixes the dimension. Any later vector of a different
+    length raises `ArgumentError`. `reset!` clears it.
+
+## 4. Predict
+
+```ruby
+posterior = predictor.predict([1.05, 2.05, 3.0])
+```
+
+You now have a distribution, not an answer:
+
+```ruby
+posterior.max_outcome           # => 1
+posterior.probability(1)        # => 0.74
+posterior.top_outcomes(3)       # => [[1, 0.74], [0, 0.15], [-1, 0.06]]
+posterior.to_h                  # => {-2 => 0.01, -1 => 0.06, 0 => 0.15, ...}
+posterior.confidence            # => 0.41
+posterior.entropy               # => 1.37
+posterior.kl_divergence_from_prior  # => 0.94
+```
+
+`puts posterior.summary` prints all of it:
+
+```
+Posterior Distribution Summary:
+================================
+Most Likely: 1 (74.0%)
+Confidence: 41.2%
+Entropy: 1.371 bits
+KL Divergence from Prior: 0.943 bits
+
+Probabilities:
+  -2: 0.010 (1.0%)
+  -1: 0.060 (6.0%)
+   0: 0.150 (15.0%)
+   1: 0.740 (74.0%)
+   2: 0.040 (4.0%)
+```
+
+## 5. Act on the uncertainty, not just the answer
+
+This is the part that makes a posterior worth having:
+
+```ruby
+if posterior.confidence < 0.25
+  # Near-uniform. The model does not know. Don't pretend otherwise.
+  :abstain
+else
+  posterior.max_outcome
+end
+```
+
+Or propagate the whole distribution downstream:
+
+```ruby
+outcomes = posterior.samples(10_000)
+outcomes.tally.sort.to_h
+# => {-2 => 98, -1 => 602, 0 => 1_503, 1 => 7_401, 2 => 396}
+```
+
+## The whole thing
+
+```ruby
+require "sqa/bi"
+
+predictor = SQA::BI.predictor(outcomes: [-2, -1, 0, 1, 2], bandwidth: 1.0)
+
+predictor.train_batch([
+  { features: [0.8, 1.9, 3.1],  outcome:  1 },
+  { features: [1.0, 2.0, 3.0],  outcome:  1 },
+  { features: [1.1, 2.1, 2.9],  outcome:  1 },
+  { features: [0.2, 0.1, 0.0],  outcome:  0 },
+  { features: [-1.0, -2.0, 0.5], outcome: -2 }
+])
+
+posterior = predictor.predict([1.05, 2.05, 3.0])
+puts posterior.summary
+```
+
+## Next
+
+- [Core Concepts](core-concepts.md) — what the three pieces actually do
+- [The Predictor](../guide/predictor.md) — the full training and prediction workflow
+- [Examples](../examples/index.md) — five runnable demos, including real market data

data/docs/guide/likelihood.md

@@ -0,0 +1,132 @@
+# The Likelihood
+
+`SQA::BI::Likelihood` estimates `P(features | outcome)` — given that an outcome
+occurred, how typical is this feature vector?
+
+This is where your data enters the calculation.
+
+## Kernel density estimation
+
+![Gaussian kernel density estimation](../assets/diagrams/kde.svg)
+
+There is no fitted model here — no coefficients, no training loop. The
+observations *are* the model. To estimate the density for an outcome:
+
+1. Take every stored observation whose outcome matches.
+2. Measure the Euclidean distance from each to the query vector.
+3. Convert each distance to a kernel value — near observations contribute a lot,
+   distant ones almost nothing.
+4. Average them.
+
+$$
+P(x \mid \text{outcome}) = \frac{1}{n} \sum_{i=1}^{n} K\!\left(\frac{\lVert x - x_i \rVert}{h}\right)
+\qquad
+K(u) = \frac{1}{\sqrt{2\pi}} e^{-u^2/2}
+$$
+
+Across all outcomes the results are normalized to sum to 1.
+
+!!! info "This is *non-parametric*, and that cuts both ways"
+    Nothing is assumed about the shape of each outcome's feature distribution —
+    it can be multimodal, skewed, whatever the data says. The cost is that every
+    prediction is O(observations), memory grows linearly with training, and
+    sparse regions of feature space give unreliable estimates.
+
+## Using it
+
+```ruby
+observations = [
+  { features: [1.0, 2.0, 3.0], outcome:  1 },
+  { features: [1.1, 2.1, 2.9], outcome:  1 },
+  { features: [-1.0, -2.0, 0.5], outcome: -2 }
+]
+
+likelihood = SQA::BI::Likelihood.new(observations, bandwidth: 1.0)
+likelihood.compute([1.0, 2.0, 3.0], [-2, -1, 0, 1, 2])
+# => {-2 => 0.05, -1 => 0.10, 0 => 0.15, 1 => 0.60, 2 => 0.10}
+```
+
+Add as you go:
+
+```ruby
+likelihood.add_observation([0.9, 1.8, 3.2], 1)
+likelihood.size            # => 4
+likelihood.outcome_counts  # => {1 => 3, -2 => 1}
+likelihood.empty?          # => false
+```
+
+`outcome_counts` is what feeds the predictor's
+[automatic prior update](prior.md#automatic-updating-inside-the-predictor).
+
+## Bandwidth
+
+`bandwidth` (`h`) sets the width of every kernel and is the single most
+consequential knob in the library.
+
+| Bandwidth | Behavior | Failure mode |
+|---|---|---|
+| Small (0.1–0.3) | Sharp, local, sensitive to fine structure | Overfits; a query far from any observation gets near-zero density everywhere |
+| Medium (0.5–1.0) | The usual working range | — |
+| Large (2.0+) | Heavily smoothed, stable | Underfits; outcomes become indistinguishable and the posterior collapses toward the prior |
+
+Changing it rebuilds the estimator from the existing observations — nothing is
+lost:
+
+```ruby
+predictor.bandwidth = 0.5
+```
+
+The right value depends on the **scale of your features**. Percentage changes
+around ±2 want a much smaller bandwidth than raw prices around 200. See
+[Tuning](tuning.md) for how to search for it honestly.
+
+## Edge cases, and what they do
+
+### No observations at all
+
+```ruby
+SQA::BI::Likelihood.new([], bandwidth: 1.0)
+           .compute([1.0, 2.0], [-1, 0, 1])
+# => {-1 => 0.333, 0 => 0.333, 1 => 0.333}
+```
+
+Uniform. With no data, the likelihood contributes nothing and the posterior is
+exactly the prior — which is the correct answer, and a useful thing to see in
+output rather than an exception.
+
+### An outcome with no observations
+
+`estimate_density` returns `1e-10` rather than `0.0`. Same reasoning as
+[Laplace smoothing on the prior](prior.md#laplace-smoothing): an exact zero is
+unrecoverable. A never-yet-observed outcome should be *very* unlikely, not
+*impossible*.
+
+### Mismatched dimensions
+
+```ruby
+# ArgumentError: Feature vectors must have same dimension
+```
+
+Raised from `euclidean_distance`. `TimeSeriesPredictor` catches this earlier
+with a clearer message, which is one reason to prefer the facade.
+
+## Performance
+
+Each `compute` is O(n · d) — every stored observation, every dimension. In
+practice that is fine at the scale this library targets: the stock demo trains
+on 5,391 observations and predicts across 1,348 test days in seconds.
+
+If you push into hundreds of thousands of observations, the honest answer is
+that plain KDE is the wrong tool and you want a tree ensemble. That comparison
+is worked through in the [exploration notes](../EXPLORATION.md).
+
+## When your evidence is text
+
+KDE needs numbers. For evidence that has no numeric form — a headline, a log
+line, an earnings call — see
+[LLM Likelihood Estimation](../llm/likelihood-estimation.md), which produces the
+same `{outcome => probability}` shape from a different mechanism.
+
+## API
+
+See [`SQA::BI::Likelihood`](../api/likelihood.md) for the full method list.

data/docs/guide/posterior.md

@@ -0,0 +1,135 @@
+# The Posterior
+
+`SQA::BI::Posterior` is the result: `P(outcome | features)`, plus everything you
+need to judge how much to trust it.
+
+## Construction
+
+```ruby
+prior       = SQA::BI::Prior.new([-2, -1, 0, 1, 2])
+likelihoods = { -2 => 0.05, -1 => 0.10, 0 => 0.15, 1 => 0.60, 2 => 0.10 }
+
+posterior = SQA::BI::Posterior.new(prior, likelihoods)
+```
+
+The computation is Bayes' theorem, done once at construction:
+
+```ruby
+unnormalized[outcome] = likelihood[outcome] * prior.probability(outcome)
+posterior = unnormalized / unnormalized.values.sum
+```
+
+`Posterior` is immutable — everything is computed up front and the object is a
+value. Normally `TimeSeriesPredictor#predict` builds it for you.
+
+!!! note "If everything multiplies out to zero"
+    When the unnormalized total is `0.0` — which can only happen if every
+    likelihood is zero for every outcome with nonzero prior — the posterior
+    falls back to the prior rather than dividing by zero. You learned nothing,
+    so your belief is unchanged. That is the right answer, not an error.
+
+## Reading the distribution
+
+```ruby
+posterior.max_outcome        # => 1                 MAP estimate
+posterior.probability(1)     # => 0.74
+posterior[1]                 # => 0.74              alias
+posterior.top_outcomes(3)    # => [[1, 0.74], [0, 0.15], [-1, 0.06]]
+posterior.to_h               # => {-2 => 0.01, ...} keyed by outcome
+posterior.to_a               # => [[-2, 0.01], ...] sorted by outcome
+```
+
+`top_outcomes` sorts by probability descending; `to_a` sorts by outcome
+ascending. The distinction matters for ordered outcomes — `to_a` preserves the
+scale, which is what you want when charting.
+
+!!! tip "Don't reach for `max_outcome` first"
+    It is the least informative thing the object knows. `1` at 74% and `1` at
+    26% are entirely different situations that `max_outcome` reports
+    identically. Read `confidence` alongside it, always.
+
+## Uncertainty
+
+```ruby
+posterior.entropy                   # => 1.371  bits
+posterior.confidence                # => 0.412
+posterior.kl_divergence_from_prior  # => 0.943  bits
+```
+
+Covered in depth in [Quantifying Uncertainty](uncertainty.md).
+
+## Sampling
+
+```ruby
+posterior.sample                       # => 1
+posterior.samples(10_000).tally.sort.to_h
+# => {-2 => 98, -1 => 602, 0 => 1_503, 1 => 7_401, 2 => 396}
+```
+
+Inverse-CDF sampling. Pass `rng:` for reproducibility:
+
+```ruby
+posterior.sample(rng: Random.new(42))
+posterior.samples(1_000, rng: Random.new(42))
+```
+
+Sampling is how you carry uncertainty into a calculation that cannot accept a
+distribution — Monte Carlo a P&L, propagate through a downstream simulation,
+generate a spread of scenarios rather than one path.
+
+## Reporting
+
+```ruby
+puts posterior.to_s
+# Posterior(-2: 0.010, -1: 0.060, 0: 0.150, 1: 0.740, 2: 0.040)
+
+puts posterior.summary
+```
+
+```
+Posterior Distribution Summary:
+================================
+Most Likely: 1 (74.0%)
+Confidence: 41.2%
+Entropy: 1.371 bits
+KL Divergence from Prior: 0.943 bits
+
+Probabilities:
+  -2: 0.010 (1.0%)
+  -1: 0.060 (6.0%)
+   0: 0.150 (15.0%)
+   1: 0.740 (74.0%)
+   2: 0.040 (4.0%)
+```
+
+## Chaining updates across several pieces of evidence
+
+A posterior can become the prior for the next update. This is how
+[LLM likelihood estimation](../llm/likelihood-estimation.md) accumulates
+evidence:
+
+```ruby
+posterior = nil
+prior = SQA::BI::Prior.new(hypotheses.keys)
+
+evidence_list.each do |evidence|
+  likelihoods = estimator.likelihoods(evidence)
+  posterior   = SQA::BI::Posterior.new(prior, likelihoods)
+  prior       = SQA::BI::Prior.new(hypotheses.keys, posterior.to_h)
+end
+```
+
+Each step's `kl_divergence_from_prior` then tells you **which evidence actually
+mattered** — in the diagnosis demo the per-update information gain ranges from
+0.025 to 0.959 bits, and the two large numbers are exactly the two facts that
+reversed the conclusion.
+
+!!! warning "Sequential updating assumes conditional independence"
+    Chaining like this treats each piece of evidence as independent given the
+    hypothesis. Two log lines that both restate the same underlying fact will
+    get counted twice and will overstate your confidence. Deduplicate evidence
+    before feeding it in — the arithmetic cannot do it for you.
+
+## API
+
+See [`SQA::BI::Posterior`](../api/posterior.md) for the full method list.