Experimental Design Checklist

Apply items to the actual design rather than treating checked boxes as a quality score. Framework versions and primary guidance: review_sources.md.

Research Question Formulation

Is the Question Well-Formed?

• [ ] Specific: Clearly defined variables and relationships
• [ ] Answerable: Can be addressed with available methods
• [ ] Relevant: Addresses a gap in knowledge or practical need
• [ ] Feasible: Resources, time, and ethical considerations allow it
• [ ] Falsifiable: Can be proven wrong if incorrect

Have You Reviewed the Literature?

• [ ] Identified what's already known
• [ ] Found gaps or contradictions to address
• [ ] Learned from methodological successes and failures
• [ ] Identified appropriate outcome measures
• [ ] Determined typical effect sizes in the field

Hypothesis Development

Is Your Hypothesis Testable?

• [ ] Makes specific, quantifiable predictions
• [ ] Variables are operationally defined
• [ ] Specifies direction/nature of expected relationships
• [ ] Can be falsified by potential observations

Types of Hypotheses

• [ ] Null hypothesis (H₀): Explicit value or range, including any non-inferiority margin
• [ ] Alternative hypothesis (H₁): Explicit competing value, range, or direction
• [ ] Directional vs. non-directional: One-tailed vs. two-tailed tests

Study Design Selection

What Type of Study is Appropriate?

Experimental (Intervention) Studies:

• [ ] Randomized Controlled Trial (RCT): Strong intervention-effect design when conduct and analysis preserve its identifying assumptions
• [ ] Quasi-experimental: Non-random assignment but manipulation
• [ ] Within-subjects: Same participants in all conditions
• [ ] Between-subjects: Different participants per condition
• [ ] Factorial: Multiple independent variables
• [ ] Crossover: Participants receive multiple interventions sequentially

Observational Studies:

• [ ] Cohort: Follow groups over time
• [ ] Case-control: Compare those with/without outcome
• [ ] Cross-sectional: Snapshot at one time point
• [ ] Ecological: Population-level data

Consider:

• [ ] Can you randomly assign participants?
• [ ] Can you manipulate the independent variable?
• [ ] Is the outcome rare (favor case-control) or common?
• [ ] Do you need to establish temporal sequence?
• [ ] What's feasible given ethical, practical constraints?

Variables

Independent Variables (Manipulated/Predictor)

• [ ] Clearly defined and operationalized
• [ ] Appropriate levels/categories chosen
• [ ] Manipulation is sufficient to test hypothesis
• [ ] Manipulation check planned (if applicable)

Dependent Variables (Outcome/Response)

• [ ] Directly measures the construct of interest
• [ ] Validated and reliable measurement
• [ ] Sensitive enough to detect expected effects
• [ ] Appropriate for statistical analysis planned
• [ ] Primary outcome clearly designated

Control Variables

• [ ] Confounding variables identified:
• Variables that affect both IV and DV
• Alternative explanations for findings
• [ ] Strategy for control:
• Randomization
• Matching
• Stratification
• Statistical adjustment
• Restriction (inclusion/exclusion criteria)
• Avoid conditioning on mediators/colliders unless required by the target estimand

Extraneous Variables

• [ ] Potential sources of noise identified
• [ ] Standardized procedures to minimize
• [ ] Environmental factors controlled
• [ ] Time of day, setting, equipment standardized

Sampling

Population Definition

• [ ] Target population: Who you want to generalize to
• [ ] Accessible population: Who you can actually sample from
• [ ] Sample: Who actually participates
• [ ] Difference between these documented

Sampling Method

• [ ] Probability sampling (preferred for generalizability):
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Systematic sampling
• [ ] Non-probability sampling (common but limits generalizability):
• Convenience sampling
• Purposive sampling
• Snowball sampling
• Quota sampling

Sample Size

• [ ] Experimental/sampling unit and independent replicate count stated (animal, donor, litter, cage, site, culture, etc.); repeated measurements do not increase independent n
• [ ] A priori power analysis conducted
• Scientifically meaningful effect and plausible variability, with sensitivity analysis
• Justified power or precision target (.80/.90 are conventions, not universal requirements)
• Significance level (typically .05)
• Statistical test to be used
• [ ] Accounts for expected attrition/dropout
• [ ] Accounts for clustering, repeated measures, and planned multiplicity
• [ ] Sufficient for planned subgroup analyses
• [ ] Practical constraints acknowledged

Inclusion/Exclusion Criteria

• [ ] Clearly defined and justified
• [ ] Not overly restrictive (limits generalizability)
• [ ] Based on theoretical or practical considerations
• [ ] Ethical considerations addressed
• [ ] Documented and applied consistently

Blinding and Randomization

Randomization

• [ ] What is randomized:
• Participant assignment to conditions
• Order of conditions (within-subjects)
• Stimuli/items presented
• [ ] Method of randomization:
• Computer-generated random numbers
• Random number tables
• Coin flips (for very small studies)
• [ ] Allocation concealment:
• Sequence generated before recruitment
• Allocation hidden until after enrollment
• Secure centralized allocation, or sequentially numbered opaque sealed envelopes with safeguards preventing anticipation or tampering
• [ ] Stratified randomization:
• Balance important variables across groups
• Block randomization to ensure equal group sizes
• [ ] Check randomization:
• Audit sequence generation and concealment; describe baseline characteristics
• Do not use baseline significance tests as proof of successful/failed randomization

Blinding

• [ ] Name who was blinded (participants, providers, assessors, analysts), to what, and how; single/double/triple-blind labels have inconsistent meanings
• [ ] Blinding feasibility:
• Is true blinding possible?
• Placebo/sham controls needed?
• Identical appearance of interventions?
• [ ] Blinding check:
• Document unblinding and its likely effect on this outcome
• Assignment guesses after treatment can reflect efficacy/adverse effects; do not interpret them automatically as a test of blinding success

Control Groups and Conditions

What Type of Control?

• [ ] No treatment control: Natural course of condition
• [ ] Placebo control: Inert treatment for comparison
• [ ] Active control: Standard treatment comparison
• [ ] Wait-list control: Delayed treatment
• [ ] Attention control: Matches contact time without active ingredient

Multiple Conditions

• [ ] Factorial designs for multiple factors
• [ ] Dose-response relationship assessment
• [ ] Mechanism testing with component analyses

Procedures

Protocol Development

• [ ] Detailed, written protocol:
• Step-by-step procedures
• Scripts for standardized instructions
• Decision rules for handling issues
• Data collection forms
• [ ] Pilot tested before main study
• [ ] Staff trained to criterion
• [ ] Compliance monitoring planned

Standardization

• [ ] Same instructions for all participants
• [ ] Same equipment and materials
• [ ] Same environment/setting when possible
• [ ] Same assessment timing
• [ ] Deviations from protocol documented

Data Collection

• [ ] When collected:
• Baseline measurements
• Post-intervention
• Follow-up timepoints
• [ ] Who collects:
• Trained researchers
• Blinded when possible
• Inter-rater reliability established
• [ ] How collected:
• Valid, reliable instruments
• Standardized administration
• Multiple methods if possible (triangulation)

Measurement

Validity

• [ ] Face validity: Appears to measure construct
• [ ] Content validity: Covers all aspects of construct
• [ ] Criterion validity: Correlates with gold standard
• Concurrent validity
• Predictive validity
• [ ] Construct validity: Measures theoretical construct
• Convergent validity (correlates with related measures)
• Discriminant validity (doesn't correlate with unrelated measures)

Reliability

• [ ] Test-retest: Consistent over time
• [ ] Internal consistency: Justify dimensionality and the chosen reliability model; Cronbach's α alone does not establish a single construct or validity
• [ ] Inter-rater reliability: Agreement between raters (Cohen's κ, ICC)
• [ ] Parallel forms: Alternative versions consistent

Measurement Considerations

• [ ] Objective measures preferred when possible
• [ ] Validated instruments used when available
• [ ] Multiple measures of key constructs
• [ ] Sensitivity to change considered
• [ ] Floor/ceiling effects avoided
• [ ] Response formats appropriate
• [ ] Recall periods appropriate
• [ ] Cultural appropriateness considered

Bias Minimization

Selection Bias

• [ ] Random sampling when possible
• [ ] Clearly defined eligibility criteria
• [ ] Document who declines and why
• [ ] Minimize self-selection

Performance Bias

• [ ] Standardized protocols
• [ ] Blinding of providers
• [ ] Monitor protocol adherence
• [ ] Document deviations

Detection Bias

• [ ] Blinding of outcome assessors
• [ ] Objective measures when possible
• [ ] Standardized assessment procedures
• [ ] Multiple raters with reliability checks

Attrition Bias

• [ ] Strategies to minimize dropout
• [ ] Track reasons for dropout
• [ ] Compare dropouts to completers
• [ ] Estimand and missing-outcome handling specified; an intention-to-treat label does not itself remove attrition bias

Reporting Bias

• [ ] Preregister study and analysis plan
• [ ] Designate primary vs. secondary outcomes
• [ ] Commit to reporting all outcomes
• [ ] Distinguish planned from exploratory analyses

Data Management

Data Collection

• [ ] Data collection forms designed and tested
• [ ] REDCap, Qualtrics, or similar platforms
• [ ] Range checks and validation rules
• [ ] Regular backups
• [ ] Secure storage and documented privacy requirements for the institution/jurisdiction

Data Quality

• [ ] Real-time data validation
• [ ] Regular quality checks
• [ ] Missing data patterns monitored
• [ ] Outliers identified and investigated
• [ ] Protocol deviations documented

Data Security

• [ ] De-identification procedures
• [ ] Access controls
• [ ] Audit trails
• [ ] Appropriate ethics-board review and applicable privacy-law requirements confirmed by the responsible institution; this checklist does not certify legal compliance

Statistical Analysis Planning

Analysis Plan (Prespecify Before Data Collection)

• [ ] Primary analysis:
• Statistical test(s) specified
• Hypothesis clearly stated
• Significance level set (usually α = .05)
• One-tailed or two-tailed
• [ ] Secondary analyses:
• Clearly designated as secondary
• Exploratory analyses labeled as such
• [ ] Multiple comparisons:
• Adjustment method specified (if needed)
• Hypothesis family covers relevant outcomes, doses, time points, subgroups, and interim analyses; naming a primary outcome alone does not control all multiplicity

Assumptions

• [ ] Assumptions of statistical tests identified
• [ ] Plan to check assumptions
• [ ] Prespecified robust or alternative analysis for the same estimand, with assumptions
• [ ] Transformation options considered

Missing Data

• [ ] Anticipated amount of missingness
• [ ] Missing data mechanism (MCAR, MAR, MNAR)
• [ ] Handling strategy:
• Complete case analysis
• Multiple imputation
• Maximum likelihood
• [ ] Sensitivity analyses planned

Effect Sizes

• [ ] Appropriate effect size measures identified
• [ ] Will be reported alongside p-values
• [ ] Confidence intervals planned

Statistical Software

• [ ] Software selected (R, SPSS, Stata, Python, etc.)
• [ ] Version documented
• [ ] Analysis scripts prepared in advance
• [ ] Will be made available (Open Science)

Ethical Considerations

Ethical Approval

• [ ] IRB/Ethics committee approval obtained
• [ ] Study registered (ClinicalTrials.gov, etc.) if applicable
• [ ] For medical research involving human participants, consult the current Declaration of Helsinki (2024 revision) and applicable local ethics requirements before the study

Informed Consent

• [ ] Voluntary participation
• [ ] Comprehensible explanation
• [ ] Risks and benefits disclosed
• [ ] Right to withdraw without penalty
• [ ] Privacy protections explained
• [ ] Compensation disclosed

Risk-Benefit Analysis

• [ ] Potential benefits outweigh risks
• [ ] Risks minimized
• [ ] Vulnerable populations protected
• [ ] Data safety monitoring (if high risk)

Confidentiality

• [ ] Data de-identified
• [ ] Secure storage
• [ ] Limited access
• [ ] Reporting doesn't allow re-identification

Validity Threats

Internal Validity (Causation)

• [ ] History: External events between measurements
• [ ] Maturation: Changes in participants over time
• [ ] Testing: Effects of repeated measurement
• [ ] Instrumentation: Changes in measurement over time
• [ ] Regression to mean: Extreme scores becoming less extreme
• [ ] Selection: Groups differ at baseline
• [ ] Attrition: Differential dropout
• [ ] Diffusion: Control group receives treatment elements

External Validity (Generalizability)

• [ ] Sample representative of population
• [ ] Setting realistic/natural
• [ ] Treatment typical of real-world implementation
• [ ] Outcome measures ecologically valid
• [ ] Time frame appropriate

Construct Validity (Measurement)

• [ ] Measures actually tap intended constructs
• [ ] Operations match theoretical definitions
• [ ] No confounding of constructs
• [ ] Adequate coverage of construct

Statistical Conclusion Validity

• [ ] Adequate statistical power
• [ ] Assumptions met
• [ ] Appropriate tests used
• [ ] Alpha level appropriate
• [ ] Multiple comparisons addressed

Reporting and Transparency

Preregistration

• [ ] Study preregistered (OSF, ClinicalTrials.gov, AsPredicted)
• [ ] Hypotheses stated a priori
• [ ] Analysis plan documented
• [ ] Distinguishes confirmatory from exploratory

Reporting Guidelines

• [ ] RCT results: CONSORT 2025 checklist (with applicable design/intervention extensions)
• [ ] Randomized trial protocols: SPIRIT 2025
• [ ] Observational studies: STROBE checklist
• [ ] Systematic reviews: PRISMA 2020 checklist
• [ ] Diagnostic accuracy studies: STARD 2015 and applicable extensions (including STARD-AI for AI diagnostic accuracy studies)
• [ ] Qualitative interviews/focus groups: COREQ checklist
• [ ] Case reports: CARE guidelines

Reporting checklists improve transparency; they are not risk-of-bias tools or certificates of valid conduct. Select extensions for the design instead of assuming every older extension has been revised to the new core checklist.

Transparency

• [ ] All measures reported
• [ ] All manipulations disclosed
• [ ] Sample size determination explained
• [ ] Exclusion criteria and numbers reported
• [ ] Attrition documented
• [ ] Deviations from protocol noted
• [ ] Conflicts of interest disclosed

Open Science

• [ ] Data sharing planned (when ethical)
• [ ] Analysis code shared
• [ ] Materials available
• [ ] Preprint posted
• [ ] Open access publication when possible

Post-Study Considerations

Data Analysis

• [ ] Follow preregistered plan
• [ ] Clearly label deviations and exploratory analyses
• [ ] Check assumptions
• [ ] Report all outcomes
• [ ] Report effect sizes and CIs, not just p-values

Interpretation

• [ ] Conclusions supported by data
• [ ] Limitations acknowledged
• [ ] Alternative explanations considered
• [ ] Generalizability discussed
• [ ] Clinical/practical significance addressed

Dissemination

• [ ] Publish regardless of results (reduce publication bias)
• [ ] Present at conferences
• [ ] Share findings with participants (when appropriate)
• [ ] Communicate to relevant stakeholders
• [ ] Plain language summaries

Next Steps

• [ ] Replication needed?
• [ ] Follow-up studies identified
• [ ] Mechanism studies planned
• [ ] Clinical applications considered

Common Pitfalls to Avoid

• [ ] No justified sample-size/precision rationale; absence of a power calculation alone does not prove that the completed study was underpowered
• [ ] Hypothesis formed after seeing data (HARKing)
• [ ] Lack of blinding with a plausible route to biased care, measurement, or analysis
• [ ] P-hacking (data fishing, optional stopping)
• [ ] Multiple testing without correction → false positives
• [ ] Inadequate control group
• [ ] Confounding not addressed
• [ ] Instruments not validated
• [ ] High attrition not addressed
• [ ] Cherry-picking results to report
• [ ] Causal language from correlational data
• [ ] Ignoring assumptions of statistical tests
• [ ] Not preregistering changes literature bias
• [ ] Conflicts of interest not disclosed

Final Checklist Before Starting

• [ ] Research question is clear and important
• [ ] Hypothesis is testable and specific
• [ ] Study design is appropriate
• [ ] Sample size is adequate (power analysis)
• [ ] Measures are valid and reliable
• [ ] Confounds are controlled
• [ ] Randomization and blinding implemented
• [ ] Data collection is standardized
• [ ] Analysis plan is prespecified
• [ ] Ethical approval obtained
• [ ] Study is preregistered
• [ ] Resources are sufficient
• [ ] Team is trained
• [ ] Protocol is documented
• [ ] Backup plans exist for problems

Remember

Good experimental design is about:

• Asking clear questions
• Minimizing bias
• Maximizing validity
• Appropriate inference
• Transparency
• Reproducibility

The best time to think about these issues is before collecting data, not after.