@pikaa-ai/pikaa 0.3.1 → 0.3.4

package/skills/scientific-critical-thinking/references/experimental_design.md

@@ -1,496 +0,0 @@
-# Experimental Design Checklist
-
-## Research Question Formulation
-
-### Is the Question Well-Formed?
-- [ ] **Specific:** Clearly defined variables and relationships
-- [ ] **Answerable:** Can be addressed with available methods
-- [ ] **Relevant:** Addresses a gap in knowledge or practical need
-- [ ] **Feasible:** Resources, time, and ethical considerations allow it
-- [ ] **Falsifiable:** Can be proven wrong if incorrect
-
-### Have You Reviewed the Literature?
-- [ ] Identified what's already known
-- [ ] Found gaps or contradictions to address
-- [ ] Learned from methodological successes and failures
-- [ ] Identified appropriate outcome measures
-- [ ] Determined typical effect sizes in the field
-
-## Hypothesis Development
-
-### Is Your Hypothesis Testable?
-- [ ] Makes specific, quantifiable predictions
-- [ ] Variables are operationally defined
-- [ ] Specifies direction/nature of expected relationships
-- [ ] Can be falsified by potential observations
-
-### Types of Hypotheses
-- [ ] **Null hypothesis (H₀):** No effect/relationship exists
-- [ ] **Alternative hypothesis (H₁):** Effect/relationship exists
-- [ ] **Directional vs. non-directional:** One-tailed vs. two-tailed tests
-
-## Study Design Selection
-
-### What Type of Study is Appropriate?
-
-**Experimental (Intervention) Studies:**
-- [ ] **Randomized Controlled Trial (RCT):** Gold standard for causation
-- [ ] **Quasi-experimental:** Non-random assignment but manipulation
-- [ ] **Within-subjects:** Same participants in all conditions
-- [ ] **Between-subjects:** Different participants per condition
-- [ ] **Factorial:** Multiple independent variables
-- [ ] **Crossover:** Participants receive multiple interventions sequentially
-
-**Observational Studies:**
-- [ ] **Cohort:** Follow groups over time
-- [ ] **Case-control:** Compare those with/without outcome
-- [ ] **Cross-sectional:** Snapshot at one time point
-- [ ] **Ecological:** Population-level data
-
-**Consider:**
-- [ ] Can you randomly assign participants?
-- [ ] Can you manipulate the independent variable?
-- [ ] Is the outcome rare (favor case-control) or common?
-- [ ] Do you need to establish temporal sequence?
-- [ ] What's feasible given ethical, practical constraints?
-
-## Variables
-
-### Independent Variables (Manipulated/Predictor)
-- [ ] Clearly defined and operationalized
-- [ ] Appropriate levels/categories chosen
-- [ ] Manipulation is sufficient to test hypothesis
-- [ ] Manipulation check planned (if applicable)
-
-### Dependent Variables (Outcome/Response)
-- [ ] Directly measures the construct of interest
-- [ ] Validated and reliable measurement
-- [ ] Sensitive enough to detect expected effects
-- [ ] Appropriate for statistical analysis planned
-- [ ] Primary outcome clearly designated
-
-### Control Variables
-- [ ] **Confounding variables identified:**
-  - Variables that affect both IV and DV
-  - Alternative explanations for findings
-- [ ] **Strategy for control:**
-  - Randomization
-  - Matching
-  - Stratification
-  - Statistical adjustment
-  - Restriction (inclusion/exclusion criteria)
-  - Blinding
-
-### Extraneous Variables
-- [ ] Potential sources of noise identified
-- [ ] Standardized procedures to minimize
-- [ ] Environmental factors controlled
-- [ ] Time of day, setting, equipment standardized
-
-## Sampling
-
-### Population Definition
-- [ ] **Target population:** Who you want to generalize to
-- [ ] **Accessible population:** Who you can actually sample from
-- [ ] **Sample:** Who actually participates
-- [ ] Difference between these documented
-
-### Sampling Method
-- [ ] **Probability sampling (preferred for generalizability):**
-  - Simple random sampling
-  - Stratified sampling
-  - Cluster sampling
-  - Systematic sampling
-- [ ] **Non-probability sampling (common but limits generalizability):**
-  - Convenience sampling
-  - Purposive sampling
-  - Snowball sampling
-  - Quota sampling
-
-### Sample Size
-- [ ] **A priori power analysis conducted**
-  - Expected effect size (from literature or pilot)
-  - Desired power (typically .80 or .90)
-  - Significance level (typically .05)
-  - Statistical test to be used
-- [ ] Accounts for expected attrition/dropout
-- [ ] Sufficient for planned subgroup analyses
-- [ ] Practical constraints acknowledged
-
-### Inclusion/Exclusion Criteria
-- [ ] Clearly defined and justified
-- [ ] Not overly restrictive (limits generalizability)
-- [ ] Based on theoretical or practical considerations
-- [ ] Ethical considerations addressed
-- [ ] Documented and applied consistently
-
-## Blinding and Randomization
-
-### Randomization
-- [ ] **What is randomized:**
-  - Participant assignment to conditions
-  - Order of conditions (within-subjects)
-  - Stimuli/items presented
-- [ ] **Method of randomization:**
-  - Computer-generated random numbers
-  - Random number tables
-  - Coin flips (for very small studies)
-- [ ] **Allocation concealment:**
-  - Sequence generated before recruitment
-  - Allocation hidden until after enrollment
-  - Sequentially numbered, sealed envelopes (if needed)
-- [ ] **Stratified randomization:**
-  - Balance important variables across groups
-  - Block randomization to ensure equal group sizes
-- [ ] **Check randomization:**
-  - Compare groups at baseline
-  - Report any significant differences
-
-### Blinding
-- [ ] **Single-blind:** Participants don't know group assignment
-- [ ] **Double-blind:** Participants and researchers don't know
-- [ ] **Triple-blind:** Participants, researchers, and data analysts don't know
-- [ ] **Blinding feasibility:**
-  - Is true blinding possible?
-  - Placebo/sham controls needed?
-  - Identical appearance of interventions?
-- [ ] **Blinding check:**
-  - Assess whether blinding maintained
-  - Ask participants/researchers to guess assignments
-
-## Control Groups and Conditions
-
-### What Type of Control?
-- [ ] **No treatment control:** Natural course of condition
-- [ ] **Placebo control:** Inert treatment for comparison
-- [ ] **Active control:** Standard treatment comparison
-- [ ] **Wait-list control:** Delayed treatment
-- [ ] **Attention control:** Matches contact time without active ingredient
-
-### Multiple Conditions
-- [ ] Factorial designs for multiple factors
-- [ ] Dose-response relationship assessment
-- [ ] Mechanism testing with component analyses
-
-## Procedures
-
-### Protocol Development
-- [ ] **Detailed, written protocol:**
-  - Step-by-step procedures
-  - Scripts for standardized instructions
-  - Decision rules for handling issues
-  - Data collection forms
-- [ ] Pilot tested before main study
-- [ ] Staff trained to criterion
-- [ ] Compliance monitoring planned
-
-### Standardization
-- [ ] Same instructions for all participants
-- [ ] Same equipment and materials
-- [ ] Same environment/setting when possible
-- [ ] Same assessment timing
-- [ ] Deviations from protocol documented
-
-### Data Collection
-- [ ] **When collected:**
-  - Baseline measurements
-  - Post-intervention
-  - Follow-up timepoints
-- [ ] **Who collects:**
-  - Trained researchers
-  - Blinded when possible
-  - Inter-rater reliability established
-- [ ] **How collected:**
-  - Valid, reliable instruments
-  - Standardized administration
-  - Multiple methods if possible (triangulation)
-
-## Measurement
-
-### Validity
-- [ ] **Face validity:** Appears to measure construct
-- [ ] **Content validity:** Covers all aspects of construct
-- [ ] **Criterion validity:** Correlates with gold standard
-  - Concurrent validity
-  - Predictive validity
-- [ ] **Construct validity:** Measures theoretical construct
-  - Convergent validity (correlates with related measures)
-  - Discriminant validity (doesn't correlate with unrelated measures)
-
-### Reliability
-- [ ] **Test-retest:** Consistent over time
-- [ ] **Internal consistency:** Items measure same construct (Cronbach's α)
-- [ ] **Inter-rater reliability:** Agreement between raters (Cohen's κ, ICC)
-- [ ] **Parallel forms:** Alternative versions consistent
-
-### Measurement Considerations
-- [ ] Objective measures preferred when possible
-- [ ] Validated instruments used when available
-- [ ] Multiple measures of key constructs
-- [ ] Sensitivity to change considered
-- [ ] Floor/ceiling effects avoided
-- [ ] Response formats appropriate
-- [ ] Recall periods appropriate
-- [ ] Cultural appropriateness considered
-
-## Bias Minimization
-
-### Selection Bias
-- [ ] Random sampling when possible
-- [ ] Clearly defined eligibility criteria
-- [ ] Document who declines and why
-- [ ] Minimize self-selection
-
-### Performance Bias
-- [ ] Standardized protocols
-- [ ] Blinding of providers
-- [ ] Monitor protocol adherence
-- [ ] Document deviations
-
-### Detection Bias
-- [ ] Blinding of outcome assessors
-- [ ] Objective measures when possible
-- [ ] Standardized assessment procedures
-- [ ] Multiple raters with reliability checks
-
-### Attrition Bias
-- [ ] Strategies to minimize dropout
-- [ ] Track reasons for dropout
-- [ ] Compare dropouts to completers
-- [ ] Intention-to-treat analysis planned
-
-### Reporting Bias
-- [ ] Preregister study and analysis plan
-- [ ] Designate primary vs. secondary outcomes
-- [ ] Commit to reporting all outcomes
-- [ ] Distinguish planned from exploratory analyses
-
-## Data Management
-
-### Data Collection
-- [ ] Data collection forms designed and tested
-- [ ] REDCap, Qualtrics, or similar platforms
-- [ ] Range checks and validation rules
-- [ ] Regular backups
-- [ ] Secure storage (HIPAA/GDPR compliant if needed)
-
-### Data Quality
-- [ ] Real-time data validation
-- [ ] Regular quality checks
-- [ ] Missing data patterns monitored
-- [ ] Outliers identified and investigated
-- [ ] Protocol deviations documented
-
-### Data Security
-- [ ] De-identification procedures
-- [ ] Access controls
-- [ ] Audit trails
-- [ ] Compliance with regulations (IRB, HIPAA, GDPR)
-
-## Statistical Analysis Planning
-
-### Analysis Plan (Prespecify Before Data Collection)
-- [ ] **Primary analysis:**
-  - Statistical test(s) specified
-  - Hypothesis clearly stated
-  - Significance level set (usually α = .05)
-  - One-tailed or two-tailed
-- [ ] **Secondary analyses:**
-  - Clearly designated as secondary
-  - Exploratory analyses labeled as such
-- [ ] **Multiple comparisons:**
-  - Adjustment method specified (if needed)
-  - Primary outcome protects from inflation
-
-### Assumptions
-- [ ] Assumptions of statistical tests identified
-- [ ] Plan to check assumptions
-- [ ] Backup non-parametric alternatives
-- [ ] Transformation options considered
-
-### Missing Data
-- [ ] Anticipated amount of missingness
-- [ ] Missing data mechanism (MCAR, MAR, MNAR)
-- [ ] Handling strategy:
-  - Complete case analysis
-  - Multiple imputation
-  - Maximum likelihood
-- [ ] Sensitivity analyses planned
-
-### Effect Sizes
-- [ ] Appropriate effect size measures identified
-- [ ] Will be reported alongside p-values
-- [ ] Confidence intervals planned
-
-### Statistical Software
-- [ ] Software selected (R, SPSS, Stata, Python, etc.)
-- [ ] Version documented
-- [ ] Analysis scripts prepared in advance
-- [ ] Will be made available (Open Science)
-
-## Ethical Considerations
-
-### Ethical Approval
-- [ ] IRB/Ethics committee approval obtained
-- [ ] Study registered (ClinicalTrials.gov, etc.) if applicable
-- [ ] Protocol follows Declaration of Helsinki or equivalent
-
-### Informed Consent
-- [ ] Voluntary participation
-- [ ] Comprehensible explanation
-- [ ] Risks and benefits disclosed
-- [ ] Right to withdraw without penalty
-- [ ] Privacy protections explained
-- [ ] Compensation disclosed
-
-### Risk-Benefit Analysis
-- [ ] Potential benefits outweigh risks
-- [ ] Risks minimized
-- [ ] Vulnerable populations protected
-- [ ] Data safety monitoring (if high risk)
-
-### Confidentiality
-- [ ] Data de-identified
-- [ ] Secure storage
-- [ ] Limited access
-- [ ] Reporting doesn't allow re-identification
-
-## Validity Threats
-
-### Internal Validity (Causation)
-- [ ] **History:** External events between measurements
-- [ ] **Maturation:** Changes in participants over time
-- [ ] **Testing:** Effects of repeated measurement
-- [ ] **Instrumentation:** Changes in measurement over time
-- [ ] **Regression to mean:** Extreme scores becoming less extreme
-- [ ] **Selection:** Groups differ at baseline
-- [ ] **Attrition:** Differential dropout
-- [ ] **Diffusion:** Control group receives treatment elements
-
-### External Validity (Generalizability)
-- [ ] Sample representative of population
-- [ ] Setting realistic/natural
-- [ ] Treatment typical of real-world implementation
-- [ ] Outcome measures ecologically valid
-- [ ] Time frame appropriate
-
-### Construct Validity (Measurement)
-- [ ] Measures actually tap intended constructs
-- [ ] Operations match theoretical definitions
-- [ ] No confounding of constructs
-- [ ] Adequate coverage of construct
-
-### Statistical Conclusion Validity
-- [ ] Adequate statistical power
-- [ ] Assumptions met
-- [ ] Appropriate tests used
-- [ ] Alpha level appropriate
-- [ ] Multiple comparisons addressed
-
-## Reporting and Transparency
-
-### Preregistration
-- [ ] Study preregistered (OSF, ClinicalTrials.gov, AsPredicted)
-- [ ] Hypotheses stated a priori
-- [ ] Analysis plan documented
-- [ ] Distinguishes confirmatory from exploratory
-
-### Reporting Guidelines
-- [ ] **RCTs:** CONSORT 2010 checklist (with applicable extensions)
-- [ ] **Observational studies:** STROBE checklist
-- [ ] **Systematic reviews:** PRISMA 2020 checklist
-- [ ] **Diagnostic studies:** STARD checklist
-- [ ] **Qualitative research:** COREQ checklist
-- [ ] **Case reports:** CARE guidelines
-
-### Transparency
-- [ ] All measures reported
-- [ ] All manipulations disclosed
-- [ ] Sample size determination explained
-- [ ] Exclusion criteria and numbers reported
-- [ ] Attrition documented
-- [ ] Deviations from protocol noted
-- [ ] Conflicts of interest disclosed
-
-### Open Science
-- [ ] Data sharing planned (when ethical)
-- [ ] Analysis code shared
-- [ ] Materials available
-- [ ] Preprint posted
-- [ ] Open access publication when possible
-
-## Post-Study Considerations
-
-### Data Analysis
-- [ ] Follow preregistered plan
-- [ ] Clearly label deviations and exploratory analyses
-- [ ] Check assumptions
-- [ ] Report all outcomes
-- [ ] Report effect sizes and CIs, not just p-values
-
-### Interpretation
-- [ ] Conclusions supported by data
-- [ ] Limitations acknowledged
-- [ ] Alternative explanations considered
-- [ ] Generalizability discussed
-- [ ] Clinical/practical significance addressed
-
-### Dissemination
-- [ ] Publish regardless of results (reduce publication bias)
-- [ ] Present at conferences
-- [ ] Share findings with participants (when appropriate)
-- [ ] Communicate to relevant stakeholders
-- [ ] Plain language summaries
-
-### Next Steps
-- [ ] Replication needed?
-- [ ] Follow-up studies identified
-- [ ] Mechanism studies planned
-- [ ] Clinical applications considered
-
-## Common Pitfalls to Avoid
-
-- [ ] No power analysis → underpowered study
-- [ ] Hypothesis formed after seeing data (HARKing)
-- [ ] No blinding when feasible → bias
-- [ ] P-hacking (data fishing, optional stopping)
-- [ ] Multiple testing without correction → false positives
-- [ ] Inadequate control group
-- [ ] Confounding not addressed
-- [ ] Instruments not validated
-- [ ] High attrition not addressed
-- [ ] Cherry-picking results to report
-- [ ] Causal language from correlational data
-- [ ] Ignoring assumptions of statistical tests
-- [ ] Not preregistering changes literature bias
-- [ ] Conflicts of interest not disclosed
-
-## Final Checklist Before Starting
-
-- [ ] Research question is clear and important
-- [ ] Hypothesis is testable and specific
-- [ ] Study design is appropriate
-- [ ] Sample size is adequate (power analysis)
-- [ ] Measures are valid and reliable
-- [ ] Confounds are controlled
-- [ ] Randomization and blinding implemented
-- [ ] Data collection is standardized
-- [ ] Analysis plan is prespecified
-- [ ] Ethical approval obtained
-- [ ] Study is preregistered
-- [ ] Resources are sufficient
-- [ ] Team is trained
-- [ ] Protocol is documented
-- [ ] Backup plans exist for problems
-
-## Remember
-
-**Good experimental design is about:**
-- Asking clear questions
-- Minimizing bias
-- Maximizing validity
-- Appropriate inference
-- Transparency
-- Reproducibility
-
-**The best time to think about these issues is before collecting data, not after.**