@synsci/cli-darwin-x64-baseline 1.1.76 → 1.1.78

package/bin/skills/statistical-analysis/references/bayesian_statistics.md

@@ -0,0 +1,661 @@
+# Bayesian Statistical Analysis
+
+This document provides guidance on conducting and interpreting Bayesian statistical analyses, which offer an alternative framework to frequentist (classical) statistics.
+
+## Bayesian vs. Frequentist Philosophy
+
+### Fundamental Differences
+
+| Aspect | Frequentist | Bayesian |
+|--------|-------------|----------|
+| **Probability interpretation** | Long-run frequency of events | Degree of belief/uncertainty |
+| **Parameters** | Fixed but unknown | Random variables with distributions |
+| **Inference** | Based on sampling distributions | Based on posterior distributions |
+| **Primary output** | p-values, confidence intervals | Posterior probabilities, credible intervals |
+| **Prior information** | Not formally incorporated | Explicitly incorporated via priors |
+| **Hypothesis testing** | Reject/fail to reject null | Probability of hypotheses given data |
+| **Sample size** | Often requires minimum | Can work with any sample size |
+| **Interpretation** | Indirect (probability of data given H₀) | Direct (probability of hypothesis given data) |
+
+### Key Question Difference
+
+**Frequentist**: "If the null hypothesis is true, what is the probability of observing data this extreme or more extreme?"
+
+**Bayesian**: "Given the observed data, what is the probability that the hypothesis is true?"
+
+The Bayesian question is more intuitive and directly addresses what researchers want to know.
+
+---
+
+## Bayes' Theorem
+
+**Formula**:
+```
+P(θ|D) = P(D|θ) × P(θ) / P(D)
+```
+
+**In words**:
+```
+Posterior = Likelihood × Prior / Evidence
+```
+
+Where:
+- **θ (theta)**: Parameter of interest (e.g., mean difference, correlation)
+- **D**: Observed data
+- **P(θ|D)**: Posterior distribution (belief about θ after seeing data)
+- **P(D|θ)**: Likelihood (probability of data given θ)
+- **P(θ)**: Prior distribution (belief about θ before seeing data)
+- **P(D)**: Marginal likelihood/evidence (normalizing constant)
+
+---
+
+## Prior Distributions
+
+### Types of Priors
+
+#### 1. Informative Priors
+
+**When to use**: When you have substantial prior knowledge from:
+- Previous studies
+- Expert knowledge
+- Theory
+- Pilot data
+
+**Example**: Meta-analysis shows effect size d ≈ 0.40, SD = 0.15
+- Prior: Normal(0.40, 0.15)
+
+**Advantages**:
+- Incorporates existing knowledge
+- More efficient (smaller samples needed)
+- Can stabilize estimates with small data
+
+**Disadvantages**:
+- Subjective (but subjectivity can be strength)
+- Must be justified and transparent
+- May be controversial if strong prior conflicts with data
+
+---
+
+#### 2. Weakly Informative Priors
+
+**When to use**: Default choice for most applications
+
+**Characteristics**:
+- Regularizes estimates (prevents extreme values)
+- Has minimal influence on posterior with moderate data
+- Prevents computational issues
+
+**Example priors**:
+- Effect size: Normal(0, 1) or Cauchy(0, 0.707)
+- Variance: Half-Cauchy(0, 1)
+- Correlation: Uniform(-1, 1) or Beta(2, 2)
+
+**Advantages**:
+- Balances objectivity and regularization
+- Computationally stable
+- Broadly acceptable
+
+---
+
+#### 3. Non-Informative (Flat/Uniform) Priors
+
+**When to use**: When attempting to be "objective"
+
+**Example**: Uniform(-∞, ∞) for any value
+
+**⚠️ Caution**:
+- Can lead to improper posteriors
+- May produce non-sensible results
+- Not truly "non-informative" (still makes assumptions)
+- Often not recommended in modern Bayesian practice
+
+**Better alternative**: Use weakly informative priors
+
+---
+
+### Prior Sensitivity Analysis
+
+**Always conduct**: Test how results change with different priors
+
+**Process**:
+1. Fit model with default/planned prior
+2. Fit model with more diffuse prior
+3. Fit model with more concentrated prior
+4. Compare posterior distributions
+
+**Reporting**:
+- If results are similar: Evidence is robust
+- If results differ substantially: Data are not strong enough to overwhelm prior
+
+**Python example**:
+```python
+import pymc as pm
+
+# Model with different priors
+priors = [
+    ('weakly_informative', pm.Normal.dist(0, 1)),
+    ('diffuse', pm.Normal.dist(0, 10)),
+    ('informative', pm.Normal.dist(0.5, 0.3))
+]
+
+results = {}
+for name, prior in priors:
+    with pm.Model():
+        effect = pm.Normal('effect', mu=prior.mu, sigma=prior.sigma)
+        # ... rest of model
+        trace = pm.sample()
+        results[name] = trace
+```
+
+---
+
+## Bayesian Hypothesis Testing
+
+### Bayes Factor (BF)
+
+**What it is**: Ratio of evidence for two competing hypotheses
+
+**Formula**:
+```
+BF₁₀ = P(D|H₁) / P(D|H₀)
+```
+
+**Interpretation**:
+
+| BF₁₀ | Evidence |
+|------|----------|
+| >100 | Decisive for H₁ |
+| 30-100 | Very strong for H₁ |
+| 10-30 | Strong for H₁ |
+| 3-10 | Moderate for H₁ |
+| 1-3 | Anecdotal for H₁ |
+| 1 | No evidence |
+| 1/3-1 | Anecdotal for H₀ |
+| 1/10-1/3 | Moderate for H₀ |
+| 1/30-1/10 | Strong for H₀ |
+| 1/100-1/30 | Very strong for H₀ |
+| <1/100 | Decisive for H₀ |
+
+**Advantages over p-values**:
+1. Can provide evidence for null hypothesis
+2. Not dependent on sampling intentions (no "peeking" problem)
+3. Directly quantifies evidence
+4. Can be updated with more data
+
+**Python calculation**:
+```python
+import pingouin as pg
+
+# Note: Limited BF support in Python
+# Better options: R packages (BayesFactor), JASP software
+
+# Approximate BF from t-statistic
+# Using Jeffreys-Zellner-Siow prior
+from scipy import stats
+
+def bf_from_t(t, n1, n2, r_scale=0.707):
+    """
+    Approximate Bayes Factor from t-statistic
+    r_scale: Cauchy prior scale (default 0.707 for medium effect)
+    """
+    # This is simplified; use dedicated packages for accurate calculation
+    df = n1 + n2 - 2
+    # Implementation requires numerical integration
+    pass
+```
+
+---
+
+### Region of Practical Equivalence (ROPE)
+
+**Purpose**: Define range of negligible effect sizes
+
+**Process**:
+1. Define ROPE (e.g., d ∈ [-0.1, 0.1] for negligible effects)
+2. Calculate % of posterior inside ROPE
+3. Make decision:
+   - >95% in ROPE: Accept practical equivalence
+   - >95% outside ROPE: Reject equivalence
+   - Otherwise: Inconclusive
+
+**Advantage**: Directly tests for practical significance
+
+**Python example**:
+```python
+# Define ROPE
+rope_lower, rope_upper = -0.1, 0.1
+
+# Calculate % of posterior in ROPE
+in_rope = np.mean((posterior_samples > rope_lower) &
+                  (posterior_samples < rope_upper))
+
+print(f"{in_rope*100:.1f}% of posterior in ROPE")
+```
+
+---
+
+## Bayesian Estimation
+
+### Credible Intervals
+
+**What it is**: Interval containing parameter with X% probability
+
+**95% Credible Interval interpretation**:
+> "There is a 95% probability that the true parameter lies in this interval."
+
+**This is what people THINK confidence intervals mean** (but don't in frequentist framework)
+
+**Types**:
+
+#### Equal-Tailed Interval (ETI)
+- 2.5th to 97.5th percentile
+- Simple to calculate
+- May not include mode for skewed distributions
+
+#### Highest Density Interval (HDI)
+- Narrowest interval containing 95% of distribution
+- Always includes mode
+- Better for skewed distributions
+
+**Python calculation**:
+```python
+import arviz as az
+
+# Equal-tailed interval
+eti = np.percentile(posterior_samples, [2.5, 97.5])
+
+# HDI
+hdi = az.hdi(posterior_samples, hdi_prob=0.95)
+```
+
+---
+
+### Posterior Distributions
+
+**Interpreting posterior distributions**:
+
+1. **Central tendency**:
+   - Mean: Average posterior value
+   - Median: 50th percentile
+   - Mode: Most probable value (MAP - Maximum A Posteriori)
+
+2. **Uncertainty**:
+   - SD: Spread of posterior
+   - Credible intervals: Quantify uncertainty
+
+3. **Shape**:
+   - Symmetric: Similar to normal
+   - Skewed: Asymmetric uncertainty
+   - Multimodal: Multiple plausible values
+
+**Visualization**:
+```python
+import matplotlib.pyplot as plt
+import arviz as az
+
+# Posterior plot with HDI
+az.plot_posterior(trace, hdi_prob=0.95)
+
+# Trace plot (check convergence)
+az.plot_trace(trace)
+
+# Forest plot (multiple parameters)
+az.plot_forest(trace)
+```
+
+---
+
+## Common Bayesian Analyses
+
+### Bayesian T-Test
+
+**Purpose**: Compare two groups (Bayesian alternative to t-test)
+
+**Outputs**:
+1. Posterior distribution of mean difference
+2. 95% credible interval
+3. Bayes Factor (BF₁₀)
+4. Probability of directional hypothesis (e.g., P(μ₁ > μ₂))
+
+**Python implementation**:
+```python
+import pymc as pm
+import arviz as az
+
+# Bayesian independent samples t-test
+with pm.Model() as model:
+    # Priors for group means
+    mu1 = pm.Normal('mu1', mu=0, sigma=10)
+    mu2 = pm.Normal('mu2', mu=0, sigma=10)
+
+    # Prior for pooled standard deviation
+    sigma = pm.HalfNormal('sigma', sigma=10)
+
+    # Likelihood
+    y1 = pm.Normal('y1', mu=mu1, sigma=sigma, observed=group1)
+    y2 = pm.Normal('y2', mu=mu2, sigma=sigma, observed=group2)
+
+    # Derived quantity: mean difference
+    diff = pm.Deterministic('diff', mu1 - mu2)
+
+    # Sample posterior
+    trace = pm.sample(2000, tune=1000, return_inferencedata=True)
+
+# Analyze results
+print(az.summary(trace, var_names=['mu1', 'mu2', 'diff']))
+
+# Probability that group1 > group2
+prob_greater = np.mean(trace.posterior['diff'].values > 0)
+print(f"P(μ₁ > μ₂) = {prob_greater:.3f}")
+
+# Plot posterior
+az.plot_posterior(trace, var_names=['diff'], ref_val=0)
+```
+
+---
+
+### Bayesian ANOVA
+
+**Purpose**: Compare three or more groups
+
+**Model**:
+```python
+import pymc as pm
+
+with pm.Model() as anova_model:
+    # Hyperpriors
+    mu_global = pm.Normal('mu_global', mu=0, sigma=10)
+    sigma_between = pm.HalfNormal('sigma_between', sigma=5)
+    sigma_within = pm.HalfNormal('sigma_within', sigma=5)
+
+    # Group means (hierarchical)
+    group_means = pm.Normal('group_means',
+                            mu=mu_global,
+                            sigma=sigma_between,
+                            shape=n_groups)
+
+    # Likelihood
+    y = pm.Normal('y',
+                  mu=group_means[group_idx],
+                  sigma=sigma_within,
+                  observed=data)
+
+    trace = pm.sample(2000, tune=1000, return_inferencedata=True)
+
+# Posterior contrasts
+contrast_1_2 = trace.posterior['group_means'][:,:,0] - trace.posterior['group_means'][:,:,1]
+```
+
+---
+
+### Bayesian Correlation
+
+**Purpose**: Estimate correlation between two variables
+
+**Advantage**: Provides distribution of correlation values
+
+**Python implementation**:
+```python
+import pymc as pm
+
+with pm.Model() as corr_model:
+    # Prior on correlation
+    rho = pm.Uniform('rho', lower=-1, upper=1)
+
+    # Convert to covariance matrix
+    cov_matrix = pm.math.stack([[1, rho],
+                                [rho, 1]])
+
+    # Likelihood (bivariate normal)
+    obs = pm.MvNormal('obs',
+                     mu=[0, 0],
+                     cov=cov_matrix,
+                     observed=np.column_stack([x, y]))
+
+    trace = pm.sample(2000, tune=1000, return_inferencedata=True)
+
+# Summarize correlation
+print(az.summary(trace, var_names=['rho']))
+
+# Probability that correlation is positive
+prob_positive = np.mean(trace.posterior['rho'].values > 0)
+```
+
+---
+
+### Bayesian Linear Regression
+
+**Purpose**: Model relationship between predictors and outcome
+
+**Advantages**:
+- Uncertainty in all parameters
+- Natural regularization (via priors)
+- Can incorporate prior knowledge
+- Credible intervals for predictions
+
+**Python implementation**:
+```python
+import pymc as pm
+
+with pm.Model() as regression_model:
+    # Priors for coefficients
+    alpha = pm.Normal('alpha', mu=0, sigma=10)  # Intercept
+    beta = pm.Normal('beta', mu=0, sigma=10, shape=n_predictors)
+    sigma = pm.HalfNormal('sigma', sigma=10)
+
+    # Expected value
+    mu = alpha + pm.math.dot(X, beta)
+
+    # Likelihood
+    y_obs = pm.Normal('y_obs', mu=mu, sigma=sigma, observed=y)
+
+    trace = pm.sample(2000, tune=1000, return_inferencedata=True)
+
+# Posterior predictive checks
+with regression_model:
+    ppc = pm.sample_posterior_predictive(trace)
+
+az.plot_ppc(ppc)
+
+# Predictions with uncertainty
+with regression_model:
+    pm.set_data({'X': X_new})
+    posterior_pred = pm.sample_posterior_predictive(trace)
+```
+
+---
+
+## Hierarchical (Multilevel) Models
+
+**When to use**:
+- Nested/clustered data (students within schools)
+- Repeated measures
+- Meta-analysis
+- Varying effects across groups
+
+**Key concept**: Partial pooling
+- Complete pooling: Ignore groups (biased)
+- No pooling: Analyze groups separately (high variance)
+- Partial pooling: Borrow strength across groups (Bayesian)
+
+**Example: Varying intercepts**:
+```python
+with pm.Model() as hierarchical_model:
+    # Hyperpriors
+    mu_global = pm.Normal('mu_global', mu=0, sigma=10)
+    sigma_between = pm.HalfNormal('sigma_between', sigma=5)
+    sigma_within = pm.HalfNormal('sigma_within', sigma=5)
+
+    # Group-level intercepts
+    alpha = pm.Normal('alpha',
+                     mu=mu_global,
+                     sigma=sigma_between,
+                     shape=n_groups)
+
+    # Likelihood
+    y_obs = pm.Normal('y_obs',
+                     mu=alpha[group_idx],
+                     sigma=sigma_within,
+                     observed=y)
+
+    trace = pm.sample()
+```
+
+---
+
+## Model Comparison
+
+### Methods
+
+#### 1. Bayes Factor
+- Directly compares model evidence
+- Sensitive to prior specification
+- Can be computationally intensive
+
+#### 2. Information Criteria
+
+**WAIC (Widely Applicable Information Criterion)**:
+- Bayesian analog of AIC
+- Lower is better
+- Accounts for effective number of parameters
+
+**LOO (Leave-One-Out Cross-Validation)**:
+- Estimates out-of-sample prediction error
+- Lower is better
+- More robust than WAIC
+
+**Python calculation**:
+```python
+import arviz as az
+
+# Calculate WAIC and LOO
+waic = az.waic(trace)
+loo = az.loo(trace)
+
+print(f"WAIC: {waic.elpd_waic:.2f}")
+print(f"LOO: {loo.elpd_loo:.2f}")
+
+# Compare multiple models
+comparison = az.compare({
+    'model1': trace1,
+    'model2': trace2,
+    'model3': trace3
+})
+print(comparison)
+```
+
+---
+
+## Checking Bayesian Models
+
+### 1. Convergence Diagnostics
+
+**R-hat (Gelman-Rubin statistic)**:
+- Compares within-chain and between-chain variance
+- Values close to 1.0 indicate convergence
+- R-hat < 1.01: Good
+- R-hat > 1.05: Poor convergence
+
+**Effective Sample Size (ESS)**:
+- Number of independent samples
+- Higher is better
+- ESS > 400 per chain recommended
+
+**Trace plots**:
+- Should look like "fuzzy caterpillar"
+- No trends, no stuck chains
+
+**Python checking**:
+```python
+# Automatic summary with diagnostics
+print(az.summary(trace, var_names=['parameter']))
+
+# Visual diagnostics
+az.plot_trace(trace)
+az.plot_rank(trace)  # Rank plots
+```
+
+---
+
+### 2. Posterior Predictive Checks
+
+**Purpose**: Does model generate data similar to observed data?
+
+**Process**:
+1. Generate predictions from posterior
+2. Compare to actual data
+3. Look for systematic discrepancies
+
+**Python implementation**:
+```python
+with model:
+    ppc = pm.sample_posterior_predictive(trace)
+
+# Visual check
+az.plot_ppc(ppc, num_pp_samples=100)
+
+# Quantitative checks
+obs_mean = np.mean(observed_data)
+pred_means = [np.mean(sample) for sample in ppc.posterior_predictive['y_obs']]
+p_value = np.mean(pred_means >= obs_mean)  # Bayesian p-value
+```
+
+---
+
+## Reporting Bayesian Results
+
+### Example T-Test Report
+
+> "A Bayesian independent samples t-test was conducted to compare groups A and B. Weakly informative priors were used: Normal(0, 1) for the mean difference and Half-Cauchy(0, 1) for the pooled standard deviation. The posterior distribution of the mean difference had a mean of 5.2 (95% CI [2.3, 8.1]), indicating that Group A scored higher than Group B. The Bayes Factor BF₁₀ = 23.5 provided strong evidence for a difference between groups, and there was a 99.7% probability that Group A's mean exceeded Group B's mean."
+
+### Example Regression Report
+
+> "A Bayesian linear regression was fitted with weakly informative priors (Normal(0, 10) for coefficients, Half-Cauchy(0, 5) for residual SD). The model explained substantial variance (R² = 0.47, 95% CI [0.38, 0.55]). Study hours (β = 0.52, 95% CI [0.38, 0.66]) and prior GPA (β = 0.31, 95% CI [0.17, 0.45]) were credible predictors (95% CIs excluded zero). Posterior predictive checks showed good model fit. Convergence diagnostics were satisfactory (all R-hat < 1.01, ESS > 1000)."
+
+---
+
+## Advantages and Limitations
+
+### Advantages
+
+1. **Intuitive interpretation**: Direct probability statements about parameters
+2. **Incorporates prior knowledge**: Uses all available information
+3. **Flexible**: Handles complex models easily
+4. **No p-hacking**: Can look at data as it arrives
+5. **Quantifies uncertainty**: Full posterior distribution
+6. **Small samples**: Works with any sample size
+
+### Limitations
+
+1. **Computational**: Requires MCMC sampling (can be slow)
+2. **Prior specification**: Requires thought and justification
+3. **Complexity**: Steeper learning curve
+4. **Software**: Fewer tools than frequentist methods
+5. **Communication**: May need to educate reviewers/readers
+
+---
+
+## Key Python Packages
+
+- **PyMC**: Full Bayesian modeling framework
+- **ArviZ**: Visualization and diagnostics
+- **Bambi**: High-level interface for regression models
+- **PyStan**: Python interface to Stan
+- **TensorFlow Probability**: Bayesian inference with TensorFlow
+
+---
+
+## When to Use Bayesian Methods
+
+**Use Bayesian when**:
+- You have prior information to incorporate
+- You want direct probability statements
+- Sample size is small
+- Model is complex (hierarchical, missing data, etc.)
+- You want to update analysis as data arrives
+
+**Frequentist may be sufficient when**:
+- Standard analysis with large sample
+- No prior information
+- Computational resources limited
+- Reviewers unfamiliar with Bayesian methods