@monoes/monomindcli 2.15.3 → 2.15.4

package/org-skills/scientific-critical-thinking/references/core_capabilities.md

@@ -0,0 +1,407 @@
+# Core Capabilities
+
+The seven capability areas in full: methodology critique, bias detection, statistical
+analysis evaluation, evidence quality assessment, logical fallacy identification,
+research design guidance, and claim evaluation — each with the questions to ask and what
+the answers imply.
+
+## Core Capabilities
+
+### 1. Methodology Critique
+
+Evaluate research methodology for rigor, validity, and potential flaws.
+
+**Apply when:**
+- Reviewing research papers
+- Assessing experimental designs
+- Evaluating study protocols
+- Planning new research
+
+**Evaluation framework:**
+
+1. **Study Design Assessment**
+   - Is the design appropriate for the research question?
+   - Can the design support causal claims being made?
+   - Are comparison groups appropriate and adequate?
+   - Consider whether experimental, quasi-experimental, or observational design is justified
+
+2. **Validity Analysis**
+   - **Internal validity:** Can we trust the causal inference?
+     - Check randomization quality
+     - Evaluate confounding control
+     - Assess selection bias
+     - Review attrition/dropout patterns
+   - **External validity:** Do results generalize?
+     - Evaluate sample representativeness
+     - Consider ecological validity of setting
+     - Assess whether conditions match target application
+   - **Construct validity:** Do measures capture intended constructs?
+     - Review measurement validation
+     - Check operational definitions
+     - Assess whether measures are direct or proxy
+   - **Statistical conclusion validity:** Are statistical inferences sound?
+     - Verify adequate power/sample size
+     - Check assumption compliance
+     - Evaluate test appropriateness
+
+3. **Control and Blinding**
+   - Was randomization properly implemented (sequence generation, allocation concealment)?
+   - Was blinding feasible and implemented (participants, providers, assessors)?
+   - Are control conditions appropriate (placebo, active control, no treatment)?
+   - Could performance or detection bias affect results?
+
+4. **Measurement Quality**
+   - Are instruments validated and reliable?
+   - Are measures objective when possible, or subjective with acknowledged limitations?
+   - Is outcome assessment standardized?
+   - Are multiple measures used to triangulate findings?
+
+**Reference:** See `references/scientific_method.md` for detailed principles and `references/experimental_design.md` for comprehensive design checklist.
+
+### 2. Bias Detection
+
+Identify and evaluate potential sources of bias that could distort findings.
+
+**Apply when:**
+- Reviewing published research
+- Designing new studies
+- Interpreting conflicting evidence
+- Assessing research quality
+
+**Systematic bias review:**
+
+1. **Cognitive Biases (Researcher)**
+   - **Confirmation bias:** Are only supporting findings highlighted?
+   - **HARKing:** Were hypotheses stated a priori or formed after seeing results?
+   - **Publication bias:** Are negative results missing from literature?
+   - **Cherry-picking:** Is evidence selectively reported?
+   - Check for preregistration and analysis plan transparency
+
+2. **Selection Biases**
+   - **Sampling bias:** Is sample representative of target population?
+   - **Volunteer bias:** Do participants self-select in systematic ways?
+   - **Attrition bias:** Is dropout differential between groups?
+   - **Survivorship bias:** Are only "survivors" visible in sample?
+   - Examine participant flow diagrams and compare baseline characteristics
+
+3. **Measurement Biases**
+   - **Observer bias:** Could expectations influence observations?
+   - **Recall bias:** Are retrospective reports systematically inaccurate?
+   - **Social desirability:** Are responses biased toward acceptability?
+   - **Instrument bias:** Do measurement tools systematically err?
+   - Evaluate blinding, validation, and measurement objectivity
+
+4. **Analysis Biases**
+   - **P-hacking:** Were multiple analyses conducted until significance emerged?
+   - **Outcome switching:** Were non-significant outcomes replaced with significant ones?
+   - **Selective reporting:** Are all planned analyses reported?
+   - **Subgroup fishing:** Were subgroup analyses conducted without correction?
+   - Check for study registration and compare to published outcomes
+
+5. **Confounding**
+   - What variables could affect both exposure and outcome?
+   - Were confounders measured and controlled (statistically or by design)?
+   - Could unmeasured confounding explain findings?
+   - Are there plausible alternative explanations?
+
+**Reference:** See `references/common_biases.md` for comprehensive bias taxonomy with detection and mitigation strategies.
+
+### 3. Statistical Analysis Evaluation
+
+Critically assess statistical methods, interpretation, and reporting.
+
+**Apply when:**
+- Reviewing quantitative research
+- Evaluating data-driven claims
+- Assessing clinical trial results
+- Reviewing meta-analyses
+
+**Statistical review checklist:**
+
+1. **Sample Size and Power**
+   - Was a priori power analysis conducted?
+   - Is sample adequate for detecting meaningful effects?
+   - Is the study underpowered (common problem)?
+   - Do significant results from small samples raise flags for inflated effect sizes?
+
+2. **Statistical Tests**
+   - Are tests appropriate for data type and distribution?
+   - Were test assumptions checked and met?
+   - Are parametric tests justified, or should non-parametric alternatives be used?
+   - Is the analysis matched to study design (e.g., paired vs. independent)?
+
+3. **Multiple Comparisons**
+   - Were multiple hypotheses tested?
+   - Was correction applied (Bonferroni, FDR, other)?
+   - Are primary outcomes distinguished from secondary/exploratory?
+   - Could findings be false positives from multiple testing?
+
+4. **P-Value Interpretation**
+   - Are p-values interpreted correctly (probability of data if null is true)?
+   - Is non-significance incorrectly interpreted as "no effect"?
+   - Is statistical significance conflated with practical importance?
+   - Are exact p-values reported, or only "p < .05"?
+   - Is there suspicious clustering just below .05?
+
+5. **Effect Sizes and Confidence Intervals**
+   - Are effect sizes reported alongside significance?
+   - Are confidence intervals provided to show precision?
+   - Is the effect size meaningful in practical terms?
+   - Are standardized effect sizes interpreted with field-specific context?
+
+6. **Missing Data**
+   - How much data is missing?
+   - Is missing data mechanism considered (MCAR, MAR, MNAR)?
+   - How is missing data handled (deletion, imputation, maximum likelihood)?
+   - Could missing data bias results?
+
+7. **Regression and Modeling**
+   - Is the model overfitted (too many predictors, no cross-validation)?
+   - Are predictions made outside the data range (extrapolation)?
+   - Are multicollinearity issues addressed?
+   - Are model assumptions checked?
+
+8. **Common Pitfalls**
+   - Correlation treated as causation
+   - Ignoring regression to the mean
+   - Base rate neglect
+   - Texas sharpshooter fallacy (pattern finding in noise)
+   - Simpson's paradox (confounding by subgroups)
+
+**Reference:** See `references/statistical_pitfalls.md` for detailed pitfalls and correct practices.
+
+### 4. Evidence Quality Assessment
+
+Evaluate the strength and quality of evidence systematically.
+
+**Apply when:**
+- Weighing evidence for decisions
+- Conducting literature reviews
+- Comparing conflicting findings
+- Determining confidence in conclusions
+
+**Evidence evaluation framework:**
+
+1. **Study Design Hierarchy**
+   - Systematic reviews/meta-analyses (highest for intervention effects)
+   - Randomized controlled trials
+   - Cohort studies
+   - Case-control studies
+   - Cross-sectional studies
+   - Case series/reports
+   - Expert opinion (lowest)
+
+   **Important:** Higher-level designs aren't always better quality. A well-designed observational study can be stronger than a poorly-conducted RCT.
+
+2. **Quality Within Design Type**
+   - Risk of bias assessment (use appropriate tool: Cochrane RoB 2 for RCTs, ROBINS-I for non-randomized studies, Newcastle-Ottawa, etc.)
+   - Methodological rigor
+   - Transparency and reporting completeness
+   - Conflicts of interest
+
+3. **GRADE Considerations (if applicable)**
+   - Start with design type (RCT = high, observational = low)
+   - **Downgrade for:**
+     - Risk of bias
+     - Inconsistency across studies
+     - Indirectness (wrong population/intervention/outcome)
+     - Imprecision (wide confidence intervals, small samples)
+     - Publication bias
+   - **Upgrade for:**
+     - Large effect sizes
+     - Dose-response relationships
+     - Confounders would reduce (not increase) effect
+
+4. **Convergence of Evidence**
+   - **Stronger when:**
+     - Multiple independent replications
+     - Different research groups and settings
+     - Different methodologies converge on same conclusion
+     - Mechanistic and empirical evidence align
+   - **Weaker when:**
+     - Single study or research group
+     - Contradictory findings in literature
+     - Publication bias evident
+     - No replication attempts
+
+5. **Contextual Factors**
+   - Biological/theoretical plausibility
+   - Consistency with established knowledge
+   - Temporality (cause precedes effect)
+   - Specificity of relationship
+   - Strength of association
+
+**Reference:** See `references/evidence_hierarchy.md` for detailed hierarchy, GRADE system, and quality assessment tools.
+
+### 5. Logical Fallacy Identification
+
+Detect and name logical errors in scientific arguments and claims.
+
+**Apply when:**
+- Evaluating scientific claims
+- Reviewing discussion/conclusion sections
+- Assessing popular science communication
+- Identifying flawed reasoning
+
+**Common fallacies in science:**
+
+1. **Causation Fallacies**
+   - **Post hoc ergo propter hoc:** "B followed A, so A caused B"
+   - **Correlation = causation:** Confusing association with causality
+   - **Reverse causation:** Mistaking cause for effect
+   - **Single cause fallacy:** Attributing complex outcomes to one factor
+
+2. **Generalization Fallacies**
+   - **Hasty generalization:** Broad conclusions from small samples
+   - **Anecdotal fallacy:** Personal stories as proof
+   - **Cherry-picking:** Selecting only supporting evidence
+   - **Ecological fallacy:** Group patterns applied to individuals
+
+3. **Authority and Source Fallacies**
+   - **Appeal to authority:** "Expert said it, so it's true" (without evidence)
+   - **Ad hominem:** Attacking person, not argument
+   - **Genetic fallacy:** Judging by origin, not merits
+   - **Appeal to nature:** "Natural = good/safe"
+
+4. **Statistical Fallacies**
+   - **Base rate neglect:** Ignoring prior probability
+   - **Texas sharpshooter:** Finding patterns in random data
+   - **Multiple comparisons:** Not correcting for multiple tests
+   - **Prosecutor's fallacy:** Confusing P(E|H) with P(H|E)
+
+5. **Structural Fallacies**
+   - **False dichotomy:** "Either A or B" when more options exist
+   - **Moving goalposts:** Changing evidence standards after they're met
+   - **Begging the question:** Circular reasoning
+   - **Straw man:** Misrepresenting arguments to attack them
+
+6. **Science-Specific Fallacies**
+   - **Galileo gambit:** "They laughed at Galileo, so my fringe idea is correct"
+   - **Argument from ignorance:** "Not proven false, so true"
+   - **Nirvana fallacy:** Rejecting imperfect solutions
+   - **Unfalsifiability:** Making untestable claims
+
+**When identifying fallacies:**
+- Name the specific fallacy
+- Explain why the reasoning is flawed
+- Identify what evidence would be needed for valid inference
+- Note that fallacious reasoning doesn't prove the conclusion false—just that this argument doesn't support it
+
+**Reference:** See `references/logical_fallacies.md` for comprehensive fallacy catalog with examples and detection strategies.
+
+### 6. Research Design Guidance
+
+Provide constructive guidance for planning rigorous studies.
+
+**Apply when:**
+- Helping design new experiments
+- Planning research projects
+- Reviewing research proposals
+- Improving study protocols
+
+**Design process:**
+
+1. **Research Question Refinement**
+   - Ensure question is specific, answerable, and falsifiable
+   - Verify it addresses a gap or contradiction in literature
+   - Confirm feasibility (resources, ethics, time)
+   - Define variables operationally
+
+2. **Design Selection**
+   - Match design to question (causal → experimental; associational → observational)
+   - Consider feasibility and ethical constraints
+   - Choose between-subjects, within-subjects, or mixed designs
+   - Plan factorial designs if testing multiple factors
+
+3. **Bias Minimization Strategy**
+   - Implement randomization when possible
+   - Plan blinding at all feasible levels (participants, providers, assessors)
+   - Identify and plan to control confounds (randomization, matching, stratification, statistical adjustment)
+   - Standardize all procedures
+   - Plan to minimize attrition
+
+4. **Sample Planning**
+   - Conduct a priori power analysis (specify expected effect, desired power, alpha)
+   - Account for attrition in sample size
+   - Define clear inclusion/exclusion criteria
+   - Consider recruitment strategy and feasibility
+   - Plan for sample representativeness
+
+5. **Measurement Strategy**
+   - Select validated, reliable instruments
+   - Use objective measures when possible
+   - Plan multiple measures of key constructs (triangulation)
+   - Ensure measures are sensitive to expected changes
+   - Establish inter-rater reliability procedures
+
+6. **Analysis Planning**
+   - Prespecify all hypotheses and analyses
+   - Designate primary outcome clearly
+   - Plan statistical tests with assumption checks
+   - Specify how missing data will be handled
+   - Plan to report effect sizes and confidence intervals
+   - Consider multiple comparison corrections
+
+7. **Transparency and Rigor**
+   - Preregister study and analysis plan
+   - Use reporting guidelines (CONSORT, STROBE, PRISMA)
+   - Plan to report all outcomes, not just significant ones
+   - Distinguish confirmatory from exploratory analyses
+   - Commit to data/code sharing
+
+**Reference:** See `references/experimental_design.md` for comprehensive design checklist covering all stages from question to dissemination.
+
+### 7. Claim Evaluation
+
+Systematically evaluate scientific claims for validity and support.
+
+**Apply when:**
+- Assessing conclusions in papers
+- Evaluating media reports of research
+- Reviewing abstract or introduction claims
+- Checking if data support conclusions
+
+**Claim evaluation process:**
+
+1. **Identify the Claim**
+   - What exactly is being claimed?
+   - Is it a causal claim, associational claim, or descriptive claim?
+   - How strong is the claim (proven, likely, suggested, possible)?
+
+2. **Assess the Evidence**
+   - What evidence is provided?
+   - Is evidence direct or indirect?
+   - Is evidence sufficient for the strength of claim?
+   - Are alternative explanations ruled out?
+
+3. **Check Logical Connection**
+   - Do conclusions follow from the data?
+   - Are there logical leaps?
+   - Is correlational data used to support causal claims?
+   - Are limitations acknowledged?
+
+4. **Evaluate Proportionality**
+   - Is confidence proportional to evidence strength?
+   - Are hedging words used appropriately?
+   - Are limitations downplayed?
+   - Is speculation clearly labeled?
+
+5. **Check for Overgeneralization**
+   - Do claims extend beyond the sample studied?
+   - Are population restrictions acknowledged?
+   - Is context-dependence recognized?
+   - Are caveats about generalization included?
+
+6. **Red Flags**
+   - Causal language from correlational studies
+   - "Proves" or absolute certainty
+   - Cherry-picked citations
+   - Ignoring contradictory evidence
+   - Dismissing limitations
+   - Extrapolation beyond data
+
+**Provide specific feedback:**
+- Quote the problematic claim
+- Explain what evidence would be needed to support it
+- Suggest appropriate hedging language if warranted
+- Distinguish between data (what was found) and interpretation (what it means)