[4eyes] great paper for HCI folks

[Cha Lee]

Folks,

Now that the quarter is over and we don't have time I actually get a
great paper I would have been excited to present at our research lunch
meetings.

Steve DiVerdi sent me a link to a great paper by Ruud Wetzels:
"Statistical Evidence in Experimental Psychology: An Empirical
Comparison Using 855 t Tests". I've attached a copy of the paper with
some of the important points highlighted.

The paper is written in a very easy to understand way, even for us
computer science folks. If you don't understand p-values or hypothesis
testing in general you should give it a read. It explains some very
fundamental concepts with regards to hypothesis testing. Effect size
is succinctly explained here without getting into the messy details.
It also introduces (to me) Bayesian measures of evidence, which seems
far more straight forward than typical p-value tests.

I'm not too sure of the back story yet but I suspect the authors were
as frustrated as I have recently been with the whole user experiments
community. Anyway, one example which was presented here was a recent
study by Dr. Bem (http://dbem.ws/FeelingFuture.pdf) where the
intention was to show that precognition exists. In 9 experiments with
over 1000 participants, 8 of 9 yielded a significant result ( 0.01 < p
< 0.05). The naive conclusion would be to point to this study and say
yes precognition exists! Using Bayesian measures the authors show that
the evidence is anecdotal and inconclusive.

In summary this paper demonstrates how very weak a p value is by
itself and that the standard 0.05 p value threshold is nonsense (as I
have begun to suspect). By looking at 855 t-tests in 2007 and
computing p values, effect sizes, and Bayes factors the authors
demonstrate that although the three different measures of statistical
significance generally agree, they do not always say the same thin
with respect to "power."

I'm no expert though so it will take a while for me to digest this but
the paper is very convincing (to me - whatever that is worth).

-- 
Regards,

Cha Lee
PhD Candidate
University of California, Santa Barbara
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Wetzelsetal2011_review.pdf
Type: application/pdf
Size: 597125 bytes
Desc: not available
URL: <https://lists.cs.ucsb.edu/pipermail/ilab-users/attachments/20120322/6b34d901/attachment-0001.pdf>