Statistical Tests#

Disclaimer: ChatGPT transformed and enhanced from stats table, did not verify full completeness and correctness yet.

Fisher’s Exact Test#

Used to evaluate associations between two categorical variables in small-sample contingency tables, especially 2×2 designs where χ² assumptions may not hold.

• Statistic: N/A
• Primary reported estimate(s): Odds ratio
• Recommended effect size: φ, Cramer’s V
• Context / descriptive statistics: Counts
• Variability: –
• Data type: Categorical, counts
• Design / dependency structure: Independent groups
• Alternative names: –
• Key assumptions: Small sample counts; independent observations
• Notes:
  • Preferred over χ² when expected cell counts are low.
  • Often used in security/HCI experiments with sparse categorical outcomes.

χ² (Chi-squared)#

Used to test whether categorical variables are associated. Common for survey analysis, demographic comparisons, and behavioral outcome frequencies.

• Statistic: χ²(df, N)
• Primary reported estimate(s): Association statistic
• Recommended effect size: φ, Cramer’s V
• Context / descriptive statistics: Counts
• Variability: –
• Data type: Categorical, counts
• Design / dependency structure: Independent groups
• Alternative names: Pearson’s χ², χ² association test, χ² test of proportion
• Key assumptions: Expected cell counts sufficiently large; independent observations
• Notes:
  • Expected cell frequencies should generally exceed 5.
  • Does not indicate direction or magnitude without an effect size.

McNemar’s χ²#

Used for paired binary outcomes, typically before/after studies or matched designs.

• Statistic: χ²(df, N)
• Primary reported estimate(s): Proportion difference, Odds ratio
• Recommended effect size: Odds ratio
• Context / descriptive statistics: Counts (2×2 paired table)
• Variability: –
• Data type: Categorical, counts
• Design / dependency structure: Paired observations
• Alternative names: –
• Key assumptions: Binary paired observations
• Notes:
  • Appropriate for repeated yes/no measurements on the same participants.

Mann-Whitney-Wilcoxon#

Nonparametric alternative to the independent t-test. Used when ordinal data or non-normal distributions make parametric assumptions questionable.

• Statistic: U, W, Z
• Primary reported estimate(s): Rank difference
• Recommended effect size: r (modified z-value)
• Context / descriptive statistics: Mean w/ SD or Median w/ IQR
• Variability: SD or IQR
• Data type: Ordinal
• Design / dependency structure: Independent groups
• Alternative names: Mann-Whitney U Test, Wilcoxon Rank-Sum Test
• Key assumptions: Independent observations; similar distribution shapes for median interpretation
• Notes:
  • Often applied to Likert-scale responses.
  • Interpreting it strictly as a “median test” is only valid under additional assumptions.

Wilcoxon Signed-Rank Test#

Nonparametric alternative to the paired t-test for paired ordinal or non-normal continuous observations.

• Statistic: W, V, Z
• Primary reported estimate(s): Paired rank difference
• Recommended effect size: r (modified z-value)
• Context / descriptive statistics: Paired differences; Median w/ IQR
• Variability: IQR
• Data type: Ordinal
• Design / dependency structure: Paired observations
• Alternative names: Pairwise Wilcoxon test, Wilcoxon Signed Rank Test, Wilcoxon-Pratt signed-rank test
• Key assumptions: Symmetric paired differences
• Notes:
  • Common in within-subject usability and user-study evaluations.

t-test#

Used to compare means between two independent groups under approximate normality assumptions.

• Statistic: t-statistic (df)
• Primary reported estimate(s): Difference in means
• Recommended effect size: Cohen’s d
• Context / descriptive statistics: Mean w/ SD
• Variability: SD
• Data type: Continuous
• Design / dependency structure: Independent groups
• Alternative names: two-sample t-test, unpaired t-test, Welch’s t-test, Student t-test
• Key assumptions: Approximate normality; independent observations
• Notes:
  • Welch’s t-test is generally preferable when variances differ.
  • Robust to moderate deviations from normality with sufficient sample size.

Paired t-test#

Used to compare means between paired or repeated observations, such as pre/post measurements.

• Statistic: t-statistic (df)
• Primary reported estimate(s): Mean paired difference
• Recommended effect size: Cohen’s d
• Context / descriptive statistics: Paired differences; Mean w/ SD
• Variability: SD
• Data type: Continuous
• Design / dependency structure: Paired observations
• Alternative names: Pairwise t-test
• Key assumptions: Approximate normality of paired differences
• Notes:
  • Appropriate when the same participants appear in both conditions.

Pearson’s Correlation#

Measures linear association strength between two continuous variables.

• Statistic: r(df)
• Primary reported estimate(s): Correlation coefficient
• Recommended effect size: Pearson’s r
• Context / descriptive statistics: Paired values
• Variability: CI optional
• Data type: Continuous
• Design / dependency structure: Paired observations
• Alternative names: Pearson Product-Moment Correlation, point-biserial correlation
• Key assumptions: Linear relationship; approximate normality
• Notes:
  • Correlation does not imply causation.
  • Sensitive to outliers.

Spearman’s Rank Correlation#

Measures monotonic association between variables using ranked observations.

• Statistic: ρ(df)
• Primary reported estimate(s): Rank correlation coefficient
• Recommended effect size: Spearman’s ρ
• Context / descriptive statistics: Paired values
• Variability: CI optional
• Data type: Ordinal
• Design / dependency structure: Paired observations
• Alternative names: Spearman’s correlation
• Key assumptions: Monotonic relationship
• Notes:
  • Frequently used for Likert-scale or ordinal survey responses. See also –> Surveys: Questionnaire for example response formats.

Linear Regression#

Models relationships between predictors and a continuous outcome variable.

• Statistic: Coefficients, t/F statistics
• Primary reported estimate(s): Regression coefficients
• Recommended effect size: Standardized coefficients optional
• Context / descriptive statistics: Predicted values, residual summaries
• Variability: CI, SE
• Data type: Continuous
• Design / dependency structure: Independent observations
• Alternative names: OLS regression, linear model, multiple regression
• Key assumptions: Linear relationship; homoscedasticity; residual normality
• Notes:
  • Regression coefficients should be interpreted in context and units.
  • Diagnostic plots are strongly recommended.

Logistic Regression#

Models binary outcomes using log-odds relationships.

• Statistic: Coefficients, z/Wald statistics
• Primary reported estimate(s): Log odds, Odds ratios, Predicted probabilities
• Recommended effect size: Odds ratio
• Context / descriptive statistics: Class counts/proportions
• Variability: CI, SE
• Data type: Binary
• Design / dependency structure: Independent observations
• Alternative names: Binomial logistic regression
• Key assumptions: Independent observations; linearity in logit
• Notes:
  • Odds ratios are generally more interpretable than raw coefficients.
  • Class imbalance should be discussed explicitly.

Ordinal Logistic Regression#

Extends logistic regression to ordered categorical outcomes.

• Statistic: Coefficients, z/Wald statistics
• Primary reported estimate(s): Log odds, Odds ratios, Predicted probabilities
• Recommended effect size: Odds ratio
• Context / descriptive statistics: Category counts/proportions
• Variability: CI, SE
• Data type: Ordinal
• Design / dependency structure: Independent observations
• Alternative names: Ordered logistic regression, proportional odds model
• Key assumptions: Proportional odds assumption
• Notes:
  • Particularly useful for Likert-scale outcomes.
  • The proportional odds assumption should be evaluated and reported.

Linear Mixed-Effects Model#

Regression framework supporting repeated measures or clustered observations through random effects.

• Statistic: Coefficients, t/F statistics
• Primary reported estimate(s): Fixed-effect coefficients
• Recommended effect size: Standardized coefficients optional
• Context / descriptive statistics: Group summaries
• Variability: CI, SE
• Data type: Continuous
• Design / dependency structure: Repeated measures / clustered
• Alternative names: Linear mixed model, multilevel linear model
• Key assumptions: Appropriate random-effects structure
• Notes:
  • Often preferable to repeated-measures ANOVA in real-world datasets.
  • Random effects should be theoretically motivated.

Mixed-Effects Logistic Regression#

Binary logistic regression with random effects for repeated or clustered observations.

• Statistic: Coefficients, z/Wald statistics
• Primary reported estimate(s): Log odds, Odds ratios, Predicted probabilities
• Recommended effect size: Odds ratio
• Context / descriptive statistics: Group/class summaries
• Variability: CI, SE
• Data type: Binary
• Design / dependency structure: Repeated measures / clustered
• Alternative names: Random-effects logistic regression, multilevel logistic model
• Key assumptions: Appropriate random-effects structure
• Notes:
  • Common in longitudinal and user-level behavioral datasets.

Mixed-Effects Ordinal Logistic Regression#

Ordinal logistic regression extended with random effects for repeated or clustered observations.

• Statistic: Coefficients, z/Wald statistics
• Primary reported estimate(s): Log odds, Odds ratios, Predicted probabilities
• Recommended effect size: Odds ratio
• Context / descriptive statistics: Group/category summaries
• Variability: CI, SE
• Data type: Ordinal
• Design / dependency structure: Repeated measures / clustered
• Alternative names: CLMM, cumulative-link mixed model
• Key assumptions: Proportional odds; appropriate random-effects structure
• Notes:
  • Increasingly common in empirical HCI and software engineering research.

ANOVA#

Used to compare means across more than two independent groups.

• Statistic: F(df₁, df₂)
• Primary reported estimate(s): Group mean differences
• Recommended effect size: ω²
• Context / descriptive statistics: Mean w/ SD
• Variability: SD
• Data type: Continuous
• Design / dependency structure: Independent groups
• Alternative names: One-Way ANOVA, Two-Way ANOVA
• Key assumptions: Approximate normality; homoscedasticity
• Notes:
  • Significant ANOVA results generally require post hoc testing.

Mixed-Effects ANOVA#

ANOVA framework for repeated-measures or within-subject experimental designs.

• Statistic: F(df₁, df₂)
• Primary reported estimate(s): Group mean differences
• Recommended effect size: Partial η²
• Context / descriptive statistics: Mean w/ SD
• Variability: SD
• Data type: Continuous
• Design / dependency structure: Repeated measures / clustered
• Alternative names: Repeated Measures ANOVA, Mixed Model ANOVA
• Key assumptions: Sphericity or appropriate correction
• Notes:
  • Linear mixed models are often more flexible alternatives.

Kruskal-Wallis Test#

Nonparametric alternative to one-way ANOVA for independent ordinal or non-normal observations.

• Statistic: H, χ²
• Primary reported estimate(s): Rank-based group difference
• Recommended effect size: η²
• Context / descriptive statistics: Median w/ IQR
• Variability: IQR
• Data type: Ordinal
• Design / dependency structure: Independent groups
• Alternative names: ANOVA on ranks, Kruskal-Wallis H test
• Key assumptions: Independent observations
• Notes:
  • Significant results typically require pairwise follow-up tests.

Friedman’s ANOVA#

Nonparametric repeated-measures alternative to ANOVA.

• Statistic: χ²(df), F
• Primary reported estimate(s): Rank-based repeated-measures difference
• Recommended effect size: Kendall’s W
• Context / descriptive statistics: Median w/ IQR
• Variability: IQR
• Data type: Ordinal
• Design / dependency structure: Repeated measures
• Alternative names: Friedman’s test
• Key assumptions: Repeated matched observations
• Notes:
  • Often used for repeated Likert-scale evaluations.