Experiments#
Resources for experiment design and analysis.
Experiments are a fundamental method in research to establish causal relationships between variables. They involve manipulating one or more independent variables (IVs) and measuring their effect on dependent variables (DVs), while controlling for confounding variables.
Common Terms#
Common Terms
Common HCS Terms:
- Participant (or User): A person who takes part in the study.
- Population: The broader group the results aim to represent.
- Sample: The group of participants included in the study, recruited from the population.
- Task: The activity participants are asked to perform (for example, logging in, identifying a phishing email).
- Measure: The specific variable recorded to assess performance or perception (for example, success rate, error rate, trust rating).
- Session: The complete period of participation for one person or group.
- Stimulus (or Prompt): The material, interface, or scenario presented to participants.
Experiment Terms
- Condition (or Treatment): A specific setup or version of the experiment that participants experience.
- Trial: A single instance or run of a task within the experiment.
- Factor: A general term for any variable that can influence results.
- Level: A specific value or category of a factor.
Variables:
- Independent Variable (IV): What the researcher changes or manipulates to test its effect.
- Dependent Variable (DV): What the researcher measures to observe the outcome of the manipulation.
- Control Variable: Something kept constant to avoid unwanted influence.
- Random Variable: A variable that introduces randomness into the experiment.
- Confounding Variable: An uncontrolled factor that may distort results.
Above Terms Used in Context#
This study investigates how different browser warning designs affect users’ decisions to avoid phishing websites.
A group of participants (or users) is recruited from a university mailing list, forming the sample that represents the broader population of general internet users. Each participant (or user) completes one session consisting of multiple tasks, such as identifying whether a displayed website is safe or suspicious.
During each trial, the stimulus (or prompt) is a simulated browser warning displayed before visiting a potentially unsafe page. The independent variable (IV) is the warning design, which has three levels: text only, color coded, and icon plus explanation. The dependent variable (DV) is click through rate, indicating how often participants proceed to the risky website. An additional measure is accuracy in labeling sites as safe or unsafe.
The control variables include device type, browser version, and task difficulty, which are held constant across all conditions (or treatments). Potential confounding variables, such as prior security training or language proficiency, are collected via a pre-study questionnaire and modeled as covariates.
Each condition (or treatment) corresponds to a distinct version of the warning, allowing comparison across the manipulated factor. Across all sessions, every participant (or user) performs 12 trials under fixed timing and presentation settings.
Types of Experiments#
Experiments in human-centered security can be categorized by the degree of control over participant assignment and experimental manipulation. The following are the primary types:
- True Experiments: Controlled assignment (e.g., random) of subjects to conditions.
- Example: Randomly give half of the participants the real drug, other half a placebo
- Quasi-Experiments: No assignment assignment, multiple groups or measures.
- Example: Grades from two different course sections
- Non-Experiments: No assignment, single group or measure.
- Example: Survey of customer satisfaction
Variables#
- Independent Variable (IV): What the researcher changes or manipulates to test its effect.
- Dependent Variable (DV): What the researcher measures to observe the outcome of the manipulation.
- Control Variable: Something kept constant to avoid unwanted influence.
- Random Variable: A variable that introduces randomness into the experiment.
- Confounding Variable: An uncontrolled factor that may distort results.
Validity in Experiments#
- Validity Trade-off
Experiment Design#
- Between-Subjects Design: Different participants in each condition.
- Pros: No carryover effects, simpler analysis.
- Cons: More participants needed, individual differences.
- Within-Subjects Design: Same participants in all conditions.
- Split-Plot Design: Combination of between- and within-subjects factors.