sqa-bi 0.1.0

data/docs/index.md

@@ -0,0 +1,110 @@
+# SQA::BI
+
+**Bayesian inference over discrete outcomes from time series data.**
+
+`SQA::BI` answers one question:
+
+> Given this feature vector, what is the probability distribution over outcomes?
+
+…where *outcomes* is a small, ordered set such as `[-2, -1, 0, 1, 2]` — read as
+<span class="outcome-down">strong downtrend</span> through
+<span class="outcome-flat">sideways</span> through
+<span class="outcome-up">strong uptrend</span>.
+
+It returns a **full posterior**, not a point estimate. Entropy, confidence and
+information gain fall out of the math rather than being bolted on afterward.
+
+!!! danger "This is a learning tool, not production software"
+    DO NOT use this library when real money is at stake. The probability
+    distributions it produces should not be taken seriously. If you lose your
+    shirt playing in the stock market, don't come crying to me. Playing in the
+    market is like playing in the street — you're going to get run over.
+
+## The shape of it
+
+![Prior × Likelihood → Posterior](assets/diagrams/bayes-pipeline.svg)
+
+```ruby
+require "sqa/bi"
+
+predictor = SQA::BI.predictor(outcomes: [-2, -1, 0, 1, 2], bandwidth: 1.0)
+
+predictor.train([1.0, 2.0, 3.0],   outcome:  1)
+predictor.train([1.1, 2.1, 2.9],   outcome:  1)
+predictor.train([-1.0, -2.0, 0.5], outcome: -2)
+
+posterior = predictor.predict([1.05, 2.05, 3.0])
+
+posterior.max_outcome    # => 1     the MAP estimate
+posterior.confidence     # => 0.41  1 − entropy / max entropy
+posterior.entropy        # => 1.37  bits
+```
+
+## Why a posterior rather than a number
+
+A classifier that says "up" tells you nothing about how close the call was. A
+posterior does:
+
+| Question | Method |
+|---|---|
+| What is most likely? | `max_outcome` |
+| How likely is each outcome? | `to_h`, `to_a`, `top_outcomes(n)` |
+| How uncertain am I? | `entropy`, `confidence` |
+| How much did this evidence teach me? | `kl_divergence_from_prior` |
+| What does the uncertainty look like downstream? | `sample`, `samples(n)` |
+
+That last row matters more than it looks. Because the belief state lives in a
+distribution rather than in prose, you can propagate it — Monte Carlo a
+portfolio, gate an action on `confidence`, or refuse to act when `entropy` is
+near its maximum.
+
+## Two ways to get a likelihood
+
+**From numbers.** [`Likelihood`](guide/likelihood.md) runs Gaussian kernel
+density estimation over the observations you have trained on. This is the
+default path and needs nothing but Ruby.
+
+**From text.** [`LlmLikelihoodEstimator`](llm/likelihood-estimation.md) asks a
+language model one narrow question per hypothesis — *assuming H is true, how
+probable is this evidence?* — and Ruby chains the updates. This is for evidence
+that has no numeric form: a news headline, a log line, a filing.
+
+The rule that governs the second path, everywhere:
+
+!!! quote "The LLM judges. Ruby computes."
+    Ask the LLM only for isolated, independent judgments — a weight, a single
+    conditional probability. Never ask it to accumulate, normalize, or update
+    beliefs. That is Bayes' job, done in Ruby.
+
+See [LLM Integration](llm/index.md) for why those two failure modes are exactly
+complementary, and [Exploration Notes](EXPLORATION.md) for the verified results.
+
+## Where it sits
+
+```
+sqa-tai  ─┐
+sqa-bi   ─┴→  sqa  →  sqa-cli / sqa-advisor / sqa-rails / sqa-sinatra
+```
+
+`sqa-bi` is a **leaf** gem with no runtime dependencies. The inference math is
+domain-agnostic — it knows nothing about markets — so `sqa` can depend on it the
+same way it depends on `sqa-tai`.
+
+The market-facing demo reaches the other direction and needs `sqa`, which is why
+it resolves through `Gemfile.local` in development only. See
+[Installation](getting-started/installation.md).
+
+## Start here
+
+<div class="grid cards" markdown>
+
+- **[Installation](getting-started/installation.md)** — add the gem, pick a bundle mode
+- **[Quick Start](getting-started/quick-start.md)** — a working predictor in twenty lines
+- **[Core Concepts](getting-started/core-concepts.md)** — outcomes, features, and what Bayes is doing
+- **[API Reference](api/index.md)** — every public method, with contracts
+
+</div>
+
+## License
+
+MIT. See [LICENSE.txt](https://github.com/MadBomber/sqa-bi/blob/main/LICENSE.txt).

data/docs/llm/index.md

@@ -0,0 +1,124 @@
+# LLM Integration
+
+## The core insight
+
+LLMs and Bayes' theorem have exactly complementary failure modes.
+
+| | Semantic judgment | Probability arithmetic |
+|---|---|---|
+| **LLM** | excellent — can assess "how surprising is this log line if the DB were down?" | poor — anchors, double-counts evidence, drifts toward its first conclusion |
+| **This library** | none — KDE needs numeric feature vectors | exact — normalization, entropy, KL divergence, sequential updates |
+
+Which yields one design rule, applied everywhere:
+
+!!! quote "The LLM judges. Ruby computes."
+    Ask the LLM only for isolated, independent judgments (a weight, a
+    conditional probability). Never ask it to accumulate, normalize, or update
+    beliefs — that is Bayes' job, done in Ruby.
+
+![LLM–Bayes loop](../assets/diagrams/llm-bayes-loop.svg)
+
+The practical consequence is that belief state lives in a `Posterior` object
+rather than in a conversation. An LLM asked to track five facts in prose
+typically anchors on whichever it saw first. Externalizing the state removes
+that failure mode entirely — not mitigates it, removes it.
+
+## Two patterns
+
+### Prior elicitation
+
+The standing objection to Bayesian methods is "where does the prior come from?"
+An LLM is a queryable compression of domain knowledge — exactly the thing a
+prior is meant to encode.
+
+[**→ Prior Elicitation**](prior-elicitation.md)
+
+```ruby
+elicitor = SQA::BI::LlmPriorElicitor.new(outcomes: [-2, -1, 0, 1, 2])
+prior = elicitor.elicit("The Fed unexpectedly cut rates by 50bp...")
+```
+
+### Likelihood estimation
+
+KDE needs numbers and history. Much real evidence is text: a log line, a news
+headline, a witness statement. An LLM can judge `P(evidence | hypothesis)` for
+that kind of evidence; Ruby chains the updates.
+
+[**→ Likelihood Estimation**](likelihood-estimation.md)
+
+```ruby
+estimator = SQA::BI::LlmLikelihoodEstimator.new(hypotheses: {...})
+estimator.likelihoods("Error rate spiked 2 minutes after deploy")
+```
+
+## Engineering guarantees
+
+These are structural, not stylistic. Preserve them in any extension.
+
+**`ruby_llm` is required lazily**, inside `LlmSupport.build_chat`. The core math
+library loads and the KDE path works without it installed. It is a development
+dependency of this gem, never a runtime one.
+
+**Both classes accept an injectable `chat:`.** The test suite passes a
+`FakeChat` and makes zero network calls. No test may reach a real provider.
+
+```ruby
+elicitor = SQA::BI::LlmPriorElicitor.new(outcomes: outcomes, chat: FakeChat.new(canned))
+```
+
+**Likelihoods are clamped to `[0.001, 0.999]`.** An overconfident model
+answering `0.0` would zero out a hypothesis permanently — no later evidence
+could ever revive it. Cromwell's rule, enforced in code.
+
+**Prompt building and response parsing are public and pure.** `build_prompt`,
+`prior_from_response`, and `likelihoods_from_response` take input and return
+output with no I/O, so prompt wording can be iterated on and tested directly.
+
+**`LlmSupport` is all `module_function`.** Every helper — JSON extraction,
+re-keying, normalization, clamping — is callable and testable in isolation.
+
+## Local first
+
+Provider resolution probes LM Studio, then Apfel, then falls back to cloud.
+That fallback is **silent by design**, which means a stopped local server looks
+exactly like a deliberate cloud run until a provider error surfaces.
+
+[**→ Providers and Local Models**](providers.md) covers the setup, the two ways
+it silently fails, and `LlmSupport.current_resolution` for making it visible.
+
+## Verified results
+
+Both patterns were validated before being documented; see
+[Exploration Notes](../EXPLORATION.md) for the full record.
+
+**Prior elicitation.** With only 5 training observations the data-only posterior
+was a dead tie between "sideways" and "mild uptrend" (0.358 each). The LLM read
+a bullish market context, produced a prior peaked at +1/+2, and the informed
+posterior broke the tie: mild uptrend at 50.0%, confidence up from 19.9% to
+30.1%.
+
+**Likelihood estimation.** A production-outage diagnosis over three hypotheses.
+After evidence 1–2, `bad_deploy` led at 78.8%. Evidence 3 ("rollback didn't
+help") and 4 ("cross-AZ ping 1ms → 900ms") cleanly reversed the belief; the
+final posterior put `network` at 99.96%. Information gain per update ranged
+0.025–0.959 bits.
+
+## Patterns not yet built
+
+Recorded in the exploration notes as future work:
+
+1. **Bayesian calibration of LLM outputs** — treat the LLM as a noisy sensor
+   with a measured confusion matrix `P(LLM says j | truth is i)`, estimated from
+   a labeled set. An LLM classification then becomes a likelihood column, and
+   Bayes yields calibrated posteriors instead of the model's overconfident
+   self-reported probabilities.
+2. **Self-consistency as sampling** — query k times at temperature > 0, treat
+   the answers as draws, update a Dirichlet posterior over the answer
+   distribution. Gives error bars on LLM judgments and a principled stopping
+   rule.
+3. **Bandwidth and hyperparameter elicitation** — let the model read a dataset
+   description and suggest bandwidth and outcome coding, closing the last manual
+   knob.
+4. **Posterior narration** — the inverse direction. Hand the LLM a posterior and
+   its KL trail and have it write the report paragraph. Zero math risk, pure
+   language work.

data/docs/llm/likelihood-estimation.md

@@ -0,0 +1,161 @@
+# Likelihood Estimation
+
+`SQA::BI::LlmLikelihoodEstimator` uses a language model as a likelihood function
+over evidence that is text rather than numbers.
+
+## The problem it solves
+
+[`Likelihood`](../guide/likelihood.md) needs numeric feature vectors and
+history. But much real-world evidence is prose: a log line, an incident report,
+a news headline, a witness statement. There is no sensible way to embed *"the
+rollback didn't help"* as a float.
+
+An LLM can judge `P(evidence | hypothesis)` for that kind of evidence. It is
+notably bad at combining several such judgments coherently — so it doesn't.
+Ruby does that part.
+
+## Usage
+
+```ruby
+estimator = SQA::BI::LlmLikelihoodEstimator.new(
+  hypotheses: {
+    bad_deploy: "The 14:02 deploy introduced a bug",
+    database:   "The primary database is degraded",
+    network:    "There is a network partition between AZs"
+  }
+)
+
+estimator.likelihoods("Error rate spiked 2 minutes after deploy")
+# => { bad_deploy: 0.9, database: 0.2, network: 0.15 }
+```
+
+!!! important "These are not probabilities of the hypotheses"
+    They need not sum to 1, and normally won't. Each answers an independent
+    question: *if this hypothesis were true, how surprising would this evidence
+    be?* Turning them into beliefs about the hypotheses is what
+    [`Posterior`](../guide/posterior.md) is for.
+
+## Chaining evidence
+
+The full pattern — each posterior becomes the next prior:
+
+```ruby
+HYPOTHESES = {
+  bad_deploy: "The 14:02 deploy introduced a bug",
+  database:   "The primary database is degraded",
+  network:    "There is a network partition between AZs"
+}
+
+estimator = SQA::BI::LlmLikelihoodEstimator.new(hypotheses: HYPOTHESES)
+prior     = SQA::BI::Prior.new(HYPOTHESES.keys)
+
+EVIDENCE.each do |evidence|
+  likelihoods = estimator.likelihoods(evidence)
+  posterior   = SQA::BI::Posterior.new(prior, likelihoods)
+
+  puts "#{evidence}"
+  puts "  information gain: #{posterior.kl_divergence_from_prior.round(3)} bits"
+  posterior.to_a.each { |id, p| puts "    #{id}: #{(p * 100).round(1)}%" }
+
+  prior = SQA::BI::Prior.new(HYPOTHESES.keys, posterior.to_h)
+end
+```
+
+`posterior_n ∝ P(evidence_n | H) × posterior_{n-1}`, computed exactly, with no
+double-counting and no anchoring drift.
+
+!!! warning "Conditional independence is assumed"
+    Chaining treats each piece of evidence as independent given the hypothesis.
+    Two log lines restating the same underlying fact get counted twice and will
+    overstate confidence. Deduplicate before feeding in — the arithmetic cannot
+    detect it.
+
+## The clamp
+
+```ruby
+def likelihoods_from_response(content)
+  raw = extract_json(content)
+  rekey_to_outcomes(raw, @hypotheses.keys)
+    .transform_values { |v| clamp_likelihood(v) }   # [0.001, 0.999]
+end
+```
+
+This is load-bearing. Without it, a model answering `0.0` for a hypothesis
+eliminates it permanently — the posterior multiplies through, and no subsequent
+evidence can revive a zero.
+
+The prompt also asks for values between 0.01 and 0.99 and says explicitly that
+evidence is rarely impossible or certain. The clamp enforces what the prompt
+requests, because prompts are not contracts.
+
+## Verified behavior
+
+From `examples/05_llm_likelihood_diagnosis.rb` — a production outage, five
+pieces of evidence arriving in order:
+
+```
+Initial belief:            bad_deploy 33.3%   database 33.3%   network 33.3%
+
+Evidence 1: HTTP 500 error rate jumped from 0.1% to 8% at 14:04, two minutes
+            after the deploy finished.
+  P(e|H): bad_deploy=0.95  database=0.60  network=0.10
+  gain: 0.351 bits         bad_deploy 57.6%   database 36.4%   network  6.1%
+...
+Evidence 5: The cloud provider posted a networking incident for our region.
+  P(e|H): bad_deploy=0.05  database=0.05  network=0.90
+  gain: 0.054 bits         bad_deploy  0.1%   database  0.1%   network 99.7%
+
+Final diagnosis: network at 99.7% confidence
+```
+
+Note the arc. Evidence 1 looks damning for the deploy — a 500 spike two minutes
+after a release is about as incriminating as observational evidence gets. But
+because likelihoods are clamped away from 0 and 1, Bayes never fully commits,
+and evidence 3–5 cleanly reverse the belief.
+
+!!! quote "This is the whole argument"
+    A single LLM asked to track the same five facts in prose typically anchors
+    on the deploy story and stays there. The belief state is not in the
+    conversation — it is in a `Posterior` object, recomputed from scratch each
+    step. There is nothing to anchor *to*.
+
+And the KL divergence tells you **which evidence mattered**: per-update
+information gain ranged 0.025–0.959 bits across the run, and the large values
+land exactly on the two facts that flipped the conclusion.
+
+## Testing without a network
+
+`build_prompt` and `likelihoods_from_response` are public and pure:
+
+```ruby
+estimator = SQA::BI::LlmLikelihoodEstimator.new(
+  hypotheses: { a: "A happened", b: "B happened" }
+)
+
+estimator.likelihoods_from_response('{"a": 0.9, "b": 0.1}')
+# => { a: 0.9, b: 0.1 }
+
+# Clamping is observable:
+estimator.likelihoods_from_response('{"a": 1.0, "b": 0.0}')
+# => { a: 0.999, b: 0.001 }
+```
+
+## Beyond diagnosis
+
+The shape generalizes past incident triage. Anywhere you have competing
+explanations and a stream of textual evidence:
+
+- **Market regime classification** — headlines and filings as evidence for
+  "risk-on" / "risk-off" / "rotation"
+- **Sentiment with a memory** — each article updates a belief instead of
+  producing an isolated score
+- **Root-cause analysis** outside software entirely
+
+The one requirement: hypotheses must be **mutually exclusive and collectively
+exhaustive**. If the real answer is "both the deploy *and* the network", a
+posterior over three single-cause hypotheses cannot represent it, and will
+confidently pick whichever one the evidence favors slightly.
+
+## API
+
+See [LLM Elicitors](../api/llm-elicitors.md) for the full method list.

data/docs/llm/prior-elicitation.md

@@ -0,0 +1,172 @@
+# Prior Elicitation
+
+`SQA::BI::LlmPriorElicitor` turns a natural-language description of a situation
+into a validated [`Prior`](../guide/prior.md).
+
+## The problem it solves
+
+Every Bayesian method runs into the same objection: *where does the prior come
+from?* The usual answers are unsatisfying — uniform (throws away everything you
+know), historical frequency (wrong exactly when the regime has changed), or
+hand-tuned (arbitrary).
+
+An LLM is a compressed, queryable archive of domain knowledge. That is precisely
+what a prior is supposed to encode.
+
+## Usage
+
+```ruby
+elicitor = SQA::BI::LlmPriorElicitor.new(
+  outcomes: [-2, -1, 0, 1, 2],
+  outcome_descriptions: {
+    -2 => "strong downtrend",
+    -1 => "mild downtrend",
+     0 => "sideways",
+     1 => "mild uptrend",
+     2 => "strong uptrend"
+  }
+)
+
+prior = elicitor.elicit(<<~CONTEXT)
+  The Federal Reserve unexpectedly cut interest rates by 50 basis points
+  yesterday. Tech-sector earnings this week broadly beat expectations.
+  However, unemployment claims ticked up slightly and consumer sentiment
+  is flat. Volatility (VIX) has dropped from 22 to 16 over five sessions.
+CONTEXT
+
+prior.probabilities
+# => {-2 => 0.0, -1 => 0.02, 0 => 0.08, 1 => 0.35, 2 => 0.55}
+```
+
+Feed it into a predictor with `update_prior: false`, or training will overwrite
+the knowledge you just injected:
+
+```ruby
+predictor = SQA::BI::TimeSeriesPredictor.new(
+  outcomes: [-2, -1, 0, 1, 2],
+  prior_probabilities: prior.probabilities,
+  update_prior: false
+)
+```
+
+!!! tip "`outcome_descriptions` is optional but not really"
+    Without labels the prompt shows bare integers, and the model has to guess
+    what `-2` means. The labels are how the semantic knowledge gets connected to
+    your outcome scale — supply them.
+
+## What actually happens
+
+The LLM is asked for **relative weights between 0 and 100**, not probabilities.
+It never normalizes, never does arithmetic. Ruby handles all of that:
+
+```ruby
+def prior_from_response(content)
+  raw     = extract_json(content)                    # find the JSON object
+  weights = rekey_to_outcomes(raw, @outcomes)        # "1" (string) → 1 (integer)
+  Prior.new(@outcomes, normalize_distribution(weights))
+end
+```
+
+| Step | Purpose |
+|---|---|
+| `extract_json` | pull a Hash out of a Hash, a raw JSON string, or prose containing a fenced JSON block |
+| `rekey_to_outcomes` | JSON keys are always strings; outcomes may be integers or symbols. Matches on `to_s`. Missing outcomes get `0.0` |
+| `normalize_distribution` | add an epsilon floor, then normalize to sum to 1 |
+| `Prior.new` | full validation — sums to 1, non-negative, complete |
+
+### The epsilon floor
+
+`normalize_distribution` adds `1e-6` to every weight before normalizing, so no
+outcome is ever assigned exactly zero.
+
+!!! warning "Cromwell's rule"
+    A zero prior can never recover, no matter how much evidence arrives — the
+    posterior multiplies prior by likelihood, and zero times anything is zero.
+    An LLM that confidently writes `"−2": 0` would otherwise permanently
+    eliminate an outcome on the strength of one prose paragraph.
+
+## Testing without a network
+
+Both `build_prompt` and `prior_from_response` are public and pure:
+
+```ruby
+elicitor = SQA::BI::LlmPriorElicitor.new(outcomes: [-1, 0, 1])
+
+prior = elicitor.prior_from_response('{"-1": 10, "0": 20, "1": 70}')
+prior.probability(1)   # => 0.7
+```
+
+Or inject a fake chat to exercise the full path:
+
+```ruby
+class FakeChat
+  Response = Struct.new(:content)
+  attr_reader :last_prompt
+
+  def initialize(content) = @content = content
+  def ask(prompt)
+    @last_prompt = prompt
+    Response.new(@content)
+  end
+end
+
+chat = FakeChat.new('{"-1": 10, "0": 20, "1": 70}')
+elicitor = SQA::BI::LlmPriorElicitor.new(outcomes: [-1, 0, 1], chat: chat)
+
+elicitor.elicit("bullish setup")
+chat.last_prompt   # assert on the prompt wording
+```
+
+## Verified behavior
+
+From the [exploration notes](../EXPLORATION.md), `examples/04_llm_elicited_prior.rb`:
+
+With only 5 training observations, the data-only posterior was a dead tie:
+
+```
+Posterior(-2: 0.000, -1: 0.139, 0: 0.358, 1: 0.358, 2: 0.144)
+MAP outcome: 1 (mild uptrend)
+Confidence:  19.9%
+```
+
+The LLM read the bullish context and returned a prior peaked at +1/+2. The
+informed posterior broke the tie:
+
+```
+Posterior(-2: 0.000, -1: 0.040, 0: 0.256, 1: 0.576, 2: 0.129)
+MAP outcome: 1 (mild uptrend)
+Confidence:  34.2%
+```
+
+**Model quality shows up as prior sharpness.** qwen3.8-27b via LM Studio gave a
+bullish prior peaked at +1; Apple's 3B on-device model was more cautious and
+peaked at 0. Neither is wrong — but the bigger model commits more, and that
+commitment is visible in the entropy.
+
+!!! note "The elicited prior correctly fades"
+    As real observations accumulate the KDE likelihood increasingly dominates
+    the product, and the informed and uniform-prior posteriors converge. The
+    prior matters most in the small-data regime — exactly where KDE is weakest
+    and you have least else to go on.
+
+## When to use it
+
+**Good fits**
+
+- **Cold start.** You have a model and almost no observations.
+- **Regime change.** The historical base rate is actively misleading.
+- **Domain knowledge with no numeric form.** "The Fed just cut rates" is not a
+  feature vector.
+
+**Poor fits**
+
+- **You have plenty of data.** The empirical base rate is better, and free.
+- **The situation is genuinely unprecedented.** The model has nothing to compress
+  and will produce a confident-sounding guess.
+- **You need auditability.** An elicited prior is not reproducible across model
+  versions. Record the exact model id and the context string alongside any
+  result you intend to defend.
+
+## API
+
+See [LLM Elicitors](../api/llm-elicitors.md) for the full method list.