[4eyes] User Study Design Note

[Cha Lee]

Hey folks,

I wanted to comment on what I perceive the role of user studies to be,
in HCI. I think there is a misunderstanding here in the lab between
when/where a user study should be conducted, how to approach it, and
generally its importance.

1)  User studies are a method to gather quantitative and qualitative
responses and are meant to be used to evaluate scientific questions
when humans are somehow in the loop. If your system/interface/question
does not require the presence of a human, then save yourself some time
and effort and skip the study.

2)  When you evaluate a system/interface with a user study, the
contribution IS THE STUDY! Generally your system isn't perfect. You
are evaluating a particular approach, not a particular system. That
is, what you are really asking is whether doing something in this
particular way is better than those other ways. This type of question
is stronger and more impactful than saying system A performed better
than system B, as it can be generalized to different instances of
systems with similar traits. I have been getting the impression that
people seem to take the user study as something of an after thought.
The study isn't something that you tack on at the end to cap off the
system you will be presenting. Granted the amount of coding is trivial
compared to the number of hours you spent working on the system... but
from an HCI standpoint, all I care about is the results from the
study. Please, please, please don't treat it as something you are
"required" to do that isn't important. If you do, then don't ask for
my help since I'm very much invested in user studies at this point.  I
seem to fight this battle every time I'm asked to help.

3)  Given this understanding, you need to have a firm understanding of
the question you are asking before you get to a study. I'm on the
extreme end, where my only goal is the user study so I spend weeks or
months making sure that I am asking a relevant question.  Don't start
the study while still unsure of what you are really asking.  Here is
user studies 101, from the idiot's (me) guide to user studies. There
are phases to user studies.  What should be reported in
academic/scientific literature (for moral/scientific correctness) is
Phase 3 study results. Phase 3 studies are studies where you determine
definitively what conditions caused what effects.  What I think
happens in every academic circle except for clinical studies, is that
Phase 2 results are reported. Phase 2 studies are studies where small
groups of participants are tested to verify how well a certain
condition works. This isn't technically correct, but hey we are
limited by budget and warm bodies. What we should stay away from is
Phase 1 or Phase 0 studies. These are exploratory studies where we
(the scientists) are not sure what the effects are at all. We are
exploring whether there is any effect, any dangers, etc. from our
manipulations or user study design. Reporting such results as... well
RESULTS, is morally incorrect as a scientist

Let me be clear: your study should only confirm or disprove your
hypothesis. If you don't have a hypothesis (a true one not something
you just made up to cross the T or dot the I) then you are not ready
for a user study.

Okay, I'm done now and I'll get off my high horse.

-- 
Regards,

Cha Lee
PhD Candidate
University of California, Santa Barbara