@brainpilot/skills 0.0.6

package/skills/02_Cross-Domain_Foundation/cogsci-statistics/SKILL.md

@@ -0,0 +1,361 @@
+---
+name: "cogsci-statistics"
+description: "Domain-specific statistical modeling guidance for cognitive science and neuroscience, encoding when and how to apply mixed models, correction methods, Bayesian approaches, and effect size reporting"
+domain: "research-methods"
+version: "1.0.0"
+papers:
+ - "Barr et al., 2013"
+ - "Baayen et al., 2008"
+ - "Benjamini & Hochberg, 1995"
+ - "Maris & Oostenveld, 2007"
+ - "Lo & Andrews, 2015"
+ - "Wagenmakers, 2007"
+dependencies:
+ required:
+ - research-literacy
+ recommended:
+ - cogsci-power-analysis
+review_status: "ai-generated"
+---
+
+# Cognitive Science Statistical Analysis
+
+## Purpose
+
+This skill encodes domain-specific statistical knowledge for cognitive science and neuroscience research. It addresses the modeling decisions, correction strategies, and reporting conventions that a general-purpose statistician or programmer would get wrong without training in the field. For concrete analysis recipes with code, see `references/common-analyses.md`.
+
+## When to Use This Skill
+
+- Choosing between repeated-measures ANOVA and mixed-effects models for a cognitive experiment
+- Specifying random effects structure for designs with subjects and items
+- Deciding how to handle reaction time (RT) data distributions
+- Selecting the appropriate multiple comparison correction
+- Deciding whether to use frequentist or Bayesian analysis
+- Reporting effect sizes and statistical results for journal submission
+
+## Research Planning Protocol
+
+Before executing the domain-specific steps below, you MUST:
+
+1. **State the research question** — What specific hypothesis is this statistical analysis testing?
+2. **Justify the method choice** — Why this statistical model? What alternatives were considered?
+3. **Declare expected outcomes** — What pattern of results would support vs. refute the hypothesis?
+4. **Note assumptions and limitations** — What does this method assume? Where could it mislead?
+5. **Present the plan to the user and WAIT for confirmation** before proceeding.
+
+For detailed methodology guidance, see the `research-literacy` skill.
+
+
+## ⚠️ Verification Notice
+
+This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please [open an issue](https://github.com/HaoxuanLiTHUAI/awesome_cognitive_and_neuroscience_skills/issues).
+
+## Repeated-Measures ANOVA vs. Mixed-Effects Models
+
+### When to Use Repeated-Measures ANOVA
+
+- Fully balanced design (no missing data, equal cell sizes)
+- Only subjects as a random factor (no item variability)
+- Simple factorial structure (2-3 factors, no continuous predictors)
+- Sphericity is met or correctable (Greenhouse-Geisser / Huynh-Feldt)
+
+### When to Use Mixed-Effects Models (LMM/GLMM)
+
+- **Crossed random effects**: Both subjects and items sampled from populations (Baayen, Clark, & Lucy, 2008; Clark, 1973). This is the norm in psycholinguistics, memory research, and any paradigm with stimulus variability.
+- Unbalanced data or missing observations
+- Continuous predictors (e.g., word frequency, stimulus duration)
+- Non-normal response distributions (RT, accuracy)
+- Need to generalize over both subjects AND items simultaneously
+
+> **Critical domain knowledge**: Clark (1973) demonstrated that failing to treat items as random effects inflates Type I error. This remains one of the most common statistical errors in cognitive science. If your stimuli are sampled from a larger population (e.g., words, faces, scenes), you must account for item variability.
+
+### Decision Logic
+
+```
+Are your stimuli sampled from a larger population?
+ |
+ +-- YES --> Mixed-effects model with crossed random effects
+ | (subjects and items)
+ |
+ +-- NO (e.g., fixed set of 4 task conditions) -->
+ |
+ +-- Any missing data, unbalanced cells, or continuous predictors?
+ | |
+ | +-- YES --> Mixed-effects model (subjects as random effect)
+ | |
+ | +-- NO --> Repeated-measures ANOVA is acceptable
+ |
+ +-- Need trial-level analysis (e.g., RT distributions)?
+ |
+ +-- YES --> Mixed-effects model (operates on individual trials)
+ +-- NO --> Repeated-measures ANOVA on condition means
+```
+
+## Random Effects Structure
+
+### The Maximal Random Effects Principle
+
+Barr et al. (2013) recommend fitting the **maximal random effects structure justified by the design** to minimize Type I error. This means including random intercepts and slopes for all within-unit factors.
+
+For a typical 2x2 design with factors A (within-subjects, within-items) and B (within-subjects, between-items):
+
+```r
+# Maximal structure (Barr et al., 2013)
+lmer(RT ~ A * B + (1 + A * B | Subject) + (1 + A | Item), data = d)
+```
+
+### When Maximal Models Fail to Converge
+
+Convergence failures are common with complex random effects. Use this hierarchy (Barr et al., 2013; Matuschek et al., 2017):
+
+1. **First**: Try a different optimizer (bobyqa, nlminb) with increased iterations (**20000** iterations; lme4 default recommendation)
+2. **Second**: Remove correlations between random effects (use `||` in lme4)
+3. **Third**: Remove the highest-order random slopes first (interaction before main effects)
+4. **Fourth**: Use a parsimonious approach guided by likelihood ratio tests (Matuschek et al., 2017)
+
+> **Do NOT** simply drop all random slopes to achieve convergence. This inflates Type I error and undermines the purpose of mixed-effects modeling (Barr et al., 2013).
+
+### Common Cognitive Science Designs and Their Random Effects
+
+| Design | Random Effects | Rationale |
+|--------|---------------|-----------|
+| Lexical decision (words as items) | `(1 + condition | subj) + (1 + condition | item)` | Subjects and words are crossed |
+| Stroop task (fixed conditions) | `(1 + congruency | subj)` | Conditions are fixed, not sampled |
+| Picture naming (pictures as items) | `(1 + SOA | subj) + (1 | item)` | Items may not vary on within-item factors |
+| Multi-site study | `(1 + condition | subj) + (1 | site)` | Site is an additional clustering factor |
+
+## Handling Reaction Time Data
+
+RT data in cognitive experiments are positively skewed, bounded below by physiological limits, and often contaminated by outliers. The approach matters.
+
+### RT Outlier Exclusion
+
+Apply these criteria **before** modeling (Ratcliff, 1993; Luce, 1986):
+
+| Criterion | Threshold | Source |
+|-----------|-----------|--------|
+| Fast outliers (anticipatory) | < **200 ms** | Whelan, 2008; Ratcliff, 1993 |
+| Slow absolute cutoff | > **2000-3000 ms** (task-dependent) | Ratcliff, 1993 |
+| Within-subject SD trimming | > **3 SD** from participant's condition mean | Van Selst & Jolicoeur, 1994 |
+| Within-subject MAD trimming | > **3 MAD** from participant's condition median | Leys et al., 2013 (more robust to skew) |
+
+> **Task-specific note**: For simple RT tasks (e.g., detection), use **100 ms** as the fast cutoff (Whelan, 2008). For choice RT tasks (e.g., lexical decision), use **200 ms** (Ratcliff, 1993). Always report exclusion rates.
+
+### RT Transformation and Modeling Strategy
+
+```
+Is your primary interest in RT distributions (not just means)?
+ |
+ +-- YES --> Drift Diffusion Model or ex-Gaussian fitting
+ |
+ +-- NO --> Choose a modeling approach:
+ |
+ +-- Option 1: Log-transform RT, then fit LMM (Gaussian)
+ | - Pro: Simple, widely understood
+ | - Con: Back-transformation of means is biased;
+ | changes the hypothesis being tested
+ | (Lo & Andrews, 2015)
+ |
+ +-- Option 2: Inverse-transform RT (1/RT = speed), then LMM
+ | - Pro: Often achieves better normality than log
+ | - Con: Same back-transformation issues as log
+ | (Ratcliff, 1993)
+ |
+ +-- Option 3 (Recommended): Generalized LMM with
+ Gamma family + identity link
+ - Pro: Models RT in original units; handles skew
+ directly; avoids transformation issues
+ (Lo & Andrews, 2015)
+ - Con: Computationally slower; may have convergence
+ issues with complex random effects
+```
+
+**Recommended default**: Gamma GLMM with identity link (Lo & Andrews, 2015). Report results on the original millisecond scale.
+
+```r
+# Recommended RT model (Lo & Andrews, 2015)
+glmer(RT ~ condition * group + (1 + condition | subj) + (1 | item),
+ family = Gamma(link = "identity"), data = d)
+```
+
+## Multiple Comparison Correction
+
+### Decision Guide for Cognitive Science
+
+| Scenario | Method | Rationale | Source |
+|----------|--------|-----------|--------|
+| Small number of planned contrasts (< 5) | **No correction** or Holm | Planned contrasts based on a priori hypotheses do not require correction if specified before data collection | Rubin, 2021 |
+| All pairwise comparisons after ANOVA | **Tukey HSD** | Controls family-wise error for all pairwise comparisons; assumes equal variance | Tukey, 1953 |
+| Many tests, correlated (e.g., EEG channels) | **Cluster-based permutation** | Respects spatial/temporal correlation structure | Maris & Oostenveld, 2007 |
+| Many tests, independent | **Bonferroni-Holm** | More powerful than Bonferroni; step-down procedure | Holm, 1979 |
+| Large-scale testing (fMRI voxels, genomics) | **FDR (Benjamini-Hochberg)** | Controls false discovery rate rather than family-wise error; appropriate when some false positives are tolerable | Benjamini & Hochberg, 1995 |
+| Exploratory whole-brain fMRI | **Cluster-level FWE** (with cluster-forming threshold **p < 0.001**) | Eklund et al. (2016) showed that **p < 0.01** cluster-forming threshold inflates false positive rates to **~70%** | Eklund et al., 2016 |
+| Confirmatory ROI analysis in fMRI | **Small volume correction (SVC)** with FWE | Restricts search space to a priori ROI | Worsley et al., 1996 |
+
+### When NOT to Correct
+
+- Single planned contrast testing a specific a priori hypothesis (Rubin, 2021)
+- Sequential Bayesian testing with BF stopping rules (evidence accumulation replaces correction; Schoenbrodt et al., 2017)
+
+## Bayesian Alternatives
+
+### When to Use Bayesian Analysis
+
+- **Quantifying evidence for the null hypothesis**: Frequentist tests cannot support H0; Bayes factors can (Wagenmakers, 2007)
+- **Small sample sizes**: Bayesian methods with informative priors can be more efficient (Kruschke, 2015)
+- **Sequential testing**: Bayes factors allow continuous monitoring without alpha inflation (Schoenbrodt et al., 2017)
+- **Complex models where p-values are unreliable**: Mixed models with small cluster sizes, or when asymptotic assumptions are questionable
+
+### Bayes Factor Interpretation
+
+| BF10 Range | Evidence Category | Source |
+|------------|------------------|--------|
+| < 1/10 | Strong evidence for H0 | Jeffreys, 1961; Lee & Wagenmakers, 2013 |
+| 1/10 to 1/3 | Moderate evidence for H0 | Lee & Wagenmakers, 2013 |
+| 1/3 to 3 | Anecdotal / inconclusive | Lee & Wagenmakers, 2013 |
+| 3 to 10 | Moderate evidence for H1 | Lee & Wagenmakers, 2013 |
+| > 10 | Strong evidence for H1 | Lee & Wagenmakers, 2013 |
+
+### Recommended Tools
+
+| Tool | Use Case | Language |
+|------|----------|----------|
+| **BayesFactor** | Standard designs (t-test, ANOVA, correlation, regression) | R |
+| **brms** | Complex models (multilevel, non-Gaussian, multivariate) | R (Stan backend) |
+| **JASP** | GUI-based Bayesian analysis for standard tests | Standalone |
+| **PyMC** | Custom Bayesian models | Python |
+
+### Reporting Bayes Factors
+
+Report the exact BF, not just the category (Wagenmakers et al., 2018):
+
+> "A Bayesian paired-samples t-test indicated moderate evidence for a difference between conditions, BF10 = 5.3 (default Cauchy prior, r = 0.707)."
+
+Always specify:
+1. The prior used (e.g., default Cauchy with scale r = **0.707** for BayesFactor t-test; Rouder et al., 2009)
+2. Direction (BF10 = evidence for H1 over H0)
+3. Robustness check: report BF across a range of prior widths
+
+## Effect Size Reporting
+
+### APA 7th Edition Requirements
+
+APA 7th edition (2020, Section 6.6) requires reporting effect sizes for all primary analyses. The specific measure depends on the test:
+
+| Test | Effect Size | Interpretation Benchmarks | Source |
+|------|------------|--------------------------|--------|
+| t-test (between groups) | **Cohen's d** | 0.2 small, 0.5 medium, 0.8 large | Cohen, 1988 |
+| t-test (within subjects) | **Cohen's d_z** or d_av | d_z uses SD of difference scores | Lakens, 2013 |
+| One-way ANOVA | **eta-squared** or **omega-squared** | 0.01 small, 0.06 medium, 0.14 large | Cohen, 1988 |
+| Factorial ANOVA | **partial eta-squared** | 0.01 small, 0.06 medium, 0.14 large | Cohen, 1988; Richardson, 2011 |
+| Mixed-effects model | **semi-partial R-squared** | No universal benchmarks; report CI | Rights & Sterba, 2019 |
+| Correlation | **r** | 0.1 small, 0.3 medium, 0.5 large | Cohen, 1988 |
+| Chi-square | **Cramer's V** or **phi** | Depends on df | Cohen, 1988 |
+
+> **Domain note**: Always report confidence intervals around effect sizes (APA 7th, 2020). Use `effectsize` (R) or `statsmodels` (Python) for computation. The benchmarks above are Cohen's generic guidelines; paradigm-specific benchmarks are more informative (see `../cogsci-power-analysis/references/effect-sizes.md`).
+
+### For Mixed-Effects Models
+
+Traditional effect sizes are not straightforward for mixed models. Options:
+
+1. **Semi-partial R-squared** via the `r2glmm` or `effectsize` package (Rights & Sterba, 2019)
+2. **Standardized regression coefficients**: Standardize predictors before fitting
+3. **Conditional and marginal R-squared**: R2m (fixed effects only) and R2c (fixed + random) via `MuMIn::r.squaredGLMM()` (Nakagawa & Schielzeth, 2013)
+
+## Common Statistical Mistakes in Cognitive Science
+
+### 1. Treating Items as Fixed Effects
+
+**Problem**: Analyzing condition means averaged over items, ignoring item variability, fails to generalize beyond the specific stimuli used (Clark, 1973).
+
+**Fix**: Use mixed-effects models with crossed random effects for subjects and items.
+
+### 2. Circular Analysis ("Double-Dipping") in Neuroimaging
+
+**Problem**: Selecting voxels/channels/time-windows based on the effect of interest, then testing that same effect (Kriegeskorte et al., 2009). Inflates effect sizes by **2x or more** (Vul et al., 2009).
+
+**Fix**: Use independent localizer, leave-one-out cross-validation, or whole-brain corrected analysis.
+
+### 3. Analyzing Accuracy with ANOVA Instead of Logistic Models
+
+**Problem**: ANOVA on proportion correct violates normality and homogeneity assumptions, especially at ceiling (> 90%) or floor (< 10%) (Jaeger, 2008; Dixon, 2008).
+
+**Fix**: Use logistic mixed-effects model on binary (correct/incorrect) trial-level data.
+
+### 4. Inappropriate Outlier Exclusion
+
+**Problem**: Removing "outlier" participants based on the dependent variable (e.g., excluding subjects whose effects go in the wrong direction) without a priori criteria.
+
+**Fix**: Define exclusion criteria before data collection. Base exclusions on performance metrics (accuracy below chance, excessive RTs), not on the effect of interest.
+
+### 5. Running ANOVAs on RT Without Addressing Skew
+
+**Problem**: ANOVA on raw RT means violates normality. Condition means conceal distributional differences (Ratcliff, 1993).
+
+**Fix**: Use Gamma GLMM (Lo & Andrews, 2015) or transform RTs, and supplement with distributional analysis if warranted.
+
+### 6. Using Uncorrected Cluster-Forming Thresholds in fMRI
+
+**Problem**: Cluster-based inference with cluster-forming thresholds more lenient than **p < 0.001** (uncorrected) produces unacceptable false positive rates up to **70%** (Eklund et al., 2016).
+
+**Fix**: Use voxel-level threshold of **p < 0.001** (uncorrected) as minimum cluster-forming threshold, or use voxel-level FWE/FDR correction.
+
+### 7. Reporting Correlation P-Values Without CIs
+
+**Problem**: A "significant" correlation of r = 0.30 with N = 50 has a 95% CI of [0.02, 0.53] -- the true effect could be near zero (Cumming, 2014).
+
+**Fix**: Always report bootstrap 95% CI for correlations. Use **10000** bootstrap samples (Efron & Tibshirani, 1993).
+
+## Minimum Statistical Reporting Checklist
+
+Based on APA 7th edition (2020) and Appelbaum et al. (2018):
+
+- [ ] Exact test statistic (F, t, chi-square, z) with degrees of freedom
+- [ ] Exact p-value (not just "< 0.05"), to 3 decimal places or "< .001"
+- [ ] Effect size with confidence interval
+- [ ] For mixed models: random effects structure, optimizer, convergence confirmation
+- [ ] For multiple comparisons: correction method and justification
+- [ ] Sample sizes for each group/condition
+- [ ] Data exclusion criteria (a priori) and proportion excluded
+- [ ] For Bayesian: prior specification, exact BF, robustness check
+
+## References
+
+- American Psychological Association. (2020). *Publication Manual of the APA* (7th ed.).
+- Appelbaum, M., et al. (2018). Journal article reporting standards for quantitative research. *American Psychologist*, 73(1), 3-25.
+- Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. *Journal of Memory and Language*, 59(4), 390-412.
+- Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing. *Journal of Memory and Language*, 68(3), 255-278.
+- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate. *Journal of the Royal Statistical Society B*, 57(1), 289-300.
+- Clark, H. H. (1973). The language-as-fixed-effect fallacy. *Journal of Verbal Learning and Verbal Behavior*, 12(4), 335-359.
+- Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Erlbaum.
+- Cumming, G. (2014). The new statistics: Why and how. *Psychological Science*, 25(1), 7-29.
+- Dixon, P. (2008). Models of accuracy in repeated-measures designs. *Journal of Memory and Language*, 59(4), 447-456.
+- Efron, B., & Tibshirani, R. J. (1993). *An Introduction to the Bootstrap*. Chapman and Hall.
+- Eklund, A., Nichols, T. E., & Knutsson, H. (2016). Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. *PNAS*, 113(28), 7900-7905.
+- Holm, S. (1979). A simple sequentially rejective multiple test procedure. *Scandinavian Journal of Statistics*, 6(2), 65-70.
+- Jaeger, T. F. (2008). Categorical data analysis: Away from ANOVAs and toward logit mixed models. *Journal of Memory and Language*, 59(4), 434-446.
+- Jeffreys, H. (1961). *Theory of Probability* (3rd ed.). Oxford University Press.
+- Kriegeskorte, N., et al. (2009). Circular analysis in systems neuroscience. *Nature Neuroscience*, 12(5), 535-540.
+- Kruschke, J. K. (2015). *Doing Bayesian Data Analysis* (2nd ed.). Academic Press.
+- Lakens, D. (2013). Calculating and reporting effect sizes. *Frontiers in Psychology*, 4, 863.
+- Lee, M. D., & Wagenmakers, E.-J. (2013). *Bayesian Cognitive Modeling*. Cambridge University Press.
+- Leys, C., et al. (2013). Detecting outliers: Do not use standard deviation around the mean. *Journal of Experimental Social Psychology*, 49(4), 764-766.
+- Lo, S., & Andrews, S. (2015). To transform or not to transform: Using generalized linear mixed models to analyse reaction time data. *Frontiers in Psychology*, 6, 1171.
+- Luce, R. D. (1986). *Response Times*. Oxford University Press.
+- Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of EEG- and MEG-data. *Journal of Neuroscience Methods*, 164(1), 177-190.
+- Matuschek, H., et al. (2017). Balancing Type I error and power in linear mixed models. *Journal of Memory and Language*, 94, 305-315.
+- Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. *Methods in Ecology and Evolution*, 4(2), 133-142.
+- Ratcliff, R. (1993). Methods for dealing with reaction time outliers. *Psychological Bulletin*, 114(3), 510-532.
+- Richardson, J. T. E. (2011). Eta squared and partial eta squared as measures of effect size. *Educational Research Review*, 6(2), 135-147.
+- Rights, J. D., & Sterba, S. K. (2019). Quantifying explained variance in multilevel models. *Journal of Educational and Behavioral Statistics*, 44(2), 223-263.
+- Rouder, J. N., et al. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. *Psychonomic Bulletin & Review*, 16(2), 225-237.
+- Rubin, M. (2021). When to adjust alpha during multiple testing. *Synthese*, 199, 10969-11000.
+- Schoenbrodt, F. D., et al. (2017). Sequential hypothesis testing with Bayes factors. *Psychological Methods*, 22(2), 322-339.
+- Van Selst, M., & Jolicoeur, P. (1994). A solution to the effect of sample size on outlier elimination. *Quarterly Journal of Experimental Psychology*, 47A(3), 631-650.
+- Vul, E., et al. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. *Perspectives on Psychological Science*, 4(3), 274-290.
+- Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. *Psychonomic Bulletin & Review*, 14(5), 779-804.
+- Wagenmakers, E.-J., et al. (2018). Bayesian inference for psychology. Part II: Example applications with JASP. *Psychonomic Bulletin & Review*, 25(1), 58-76.
+- Whelan, R. (2008). Effective analysis of reaction time data. *The Psychological Record*, 58(3), 475-482.
+- Worsley, K. J., et al. (1996). A unified statistical approach for determining significant signals in images of cerebral activation. *Human Brain Mapping*, 4(1), 58-73.
+
+See `references/common-analyses.md` for concrete analysis recipes with code patterns.