A smart paper about empirical study of ubiquitous computing: It's a jungle out there:Â practical considerations for evaluation in the city by Melanie Kellar, Derek Reilly, Kirstie Hawkey, Malcolm Rodgers, Bonnie MacKay, David Dearman, Vicki Ha, W. Joseph MacInnes*, Michael Nunes, Karen Parker, Tara Whalen, Kori M. Inkpen (tons of writers!):
An essential aspect of mobile and ubiquitous computing research is evaluation within the expected usage context, including environment. When that environment is an urban center, it can be dynamic, expansive, and unpredictable. Methodologies that focus on genuine use in the environment can uncover valuable insights, although they may also limit measurement and control. In this paper, we present our experiences applying traditional experimental techniques for field research in two separate projects set in urban environments. We argue that although traditional methods may be difficult to apply in cities, the challenges are surmountable, and this kind of field research can be a crucial component of evaluation.
Why do I blog this? Though short, this paper is very relevant with what we do here at the lab (+future company). The paper gives two nice field studies the authors conducted and then elaborate on the issues they had to face (collecting data, bias, noises...). The 'observation in context' is now more and more integrated which is a good point. What is also clever here is the fact that they reflect on one of the experiment failure, and report on what they learnt from this. The conclusion is somehow close to what we got from CatchBob! (translated from a campus environment in our case)
We have reflected upon our experiences applying standard field research techniques when evaluating technology in an urban context. Dynamic and unpredictable, urban environments seriously challenge experimental observation and control. Yet, as our experiences demonstrate, there are also tremendous insights to be gained.
I am also interested in this reference they quote: McGrath, J. E. (1995). Methodology Matters: Doing Research in the Behavioral and Social Sciences. In S. Greenberg (Ed.), Human-Computer Interaction: Toward the Year 2000 (pp. 152-169).