Recently working on a project about gestural interfaces and the user experience of the Nintendo Wii, I had my share of discussions about sampling in user experience research and the role of exceptions. Quantitative researchers often drawn nice curves with cute statistical distributions with "means" and quantiles. The type of things I've done in my PhD research, measuring X and Z (satisfaction to a certain project, number of messages typed on a phone, number of time someone pressed a certain button, etc.). In the end, you get this sort of graph represented below with anonymized dots which eventually represents how normal humans did certain things.
In general, quant research (the sort I've done in the past yes) compares different "conditions": you have two sorts of interfaces, each group of users test one of the interface and you compare the number of time a certain group did certain things on the interface they had. Say, the number of time they pressed on the button called "OK". Applying different statistical techniques (like variance analysis is the distribution is normal in the statistical sense, checking variances and if you're in trouble then you always employ "non-parametric tests"). This is robust no kidding, I don't criticize that kind of method. However, what I am wondering about is when this sort of methodology is solely applied to design research.
And it leads me to the discussion I had the other day with a colleague about the importance of exceptions, dots which are not close to the means, the weird outliers, peeps who do not fall in the distribution like that weird circle on the upper right-hand corner on the boxplot below:
Depending on your mood, the research methodology and your colleagues' attitude, there is a wide spectrum of reaction ranging from "WTF, that person screwing my distribution?" to "OK this is an extreme user, he/she is special, let's have a look more closely". And then, of course, because you're a smarty pant and you ALSO have qualitative data you see what the person SAID or DID (or whatever other types of data sources you have). Then the real thing starts: who are the extreme users? how extreme are they? what makes them extreme? are there other data source which attest that they are "exceptions". And obviously this leads you to the question the norm (the mean).
To some extent, that's the story of why I slowly moved from quant research to a mix of descriptive quantitative and qualitative research in user experience projects. I started getting interested in the role of exceptions, especially with regards to their importance in design. Why exceptions are important in design? Perhaps because they might show peculiar behavior and routine which can announce futures norms or trends (and then inspire new products, features and services) but also to show that the notion of a "normal user" or "mean user" is difficult to grasp as diversity exist and is important. Surely a very relevant near future laboratory spin.
An interesting example of an extreme user was this deaf guy I saw the other day at the train station, walking and gesticulating in front of his video cell-phone. If you map the use of video-communication on cell-phone you get a very low usage of the feature in general but that guy would be an exception.