User Study Notes

Marchionini 357/fall04

Research question

Approach

Evaluate single system

Compare variations (distinct, disabled, control)

Treatments/System implementation(s) and relationship to RQ.

Validity?

Dependent variables (what to measure? how to capture?)

Time to completion

Accuracy

Errors

Satisfaction/preferences

Paths/patterns

Tasks

Validity with respect to RQ?

Reliability with respect to dependent variables?

Number of tasks/repetitions

Participants (subjects)

Number

Homogeneity/categorizations (e.g., expertise, experience, demographics)

Motivation

Procedures

Individuals vs groups?

recruitment

Order effects & learning effects (tasks, subjects, treatments)

Logistics

IRB and consent forms

Lab setup (or laptop, etc.)

Scheduling

Payments

Confidentiality

Training? Training in use (develop adequacy, reduce novelty effects, but not create bias)

Instructions and Web-based control/stimulus [could be enormous effort]

Data collection (audio, video, notes, questionnaires, logs, etc.)

Time limits ? Reflections after each task? Debriefing ?

Think aloud? Stimulated recall?

Analysis

Statistical summaries

Statistical comparisons (which statistical tests? Data groups? Combining/disaggregating data)

Qualitative summaries (e.g., full transcripts or summaries ?)

Writing

Venue (conference, journal, tech report)

Organization?

Data to include (appendices?)

Meta factors

PILOT testing!

Relationship to larger research agenda

Relationship to larger system development