The 2005 UbiApp Workshop: What Makes Good Application-Led Research? by Richard Sharp and Kasim Rehman, IEEE pervasive computing, Vol. 4, No. 3, July–Sept. 2005 The paper is a summary of a workshop that happened in 2005 at the Ubicomp conference. It deals with the concept of "application-led" research: projects that aim at designing, implementing, deploying and evaluating applications using an interdisciplinary approach (computer sciences, social sciences, ethnography, HCI). And it's motivated by current world's problems. This corresponds to the distinction made by Järvinen about social sciences: "Researches stressing utility of innovations" versus "Researches stressing what is reality" (understanding a behavior in psychology for instance). These 2 categories are of course different form the technology-led research which is interested in developing applications for a pure technical perspective. The point if this workshop was to discuss what are the criteria to judge this type of research.
A consensus people reached at this workshop was that "the ubiquitous computing community to be effective, it must engage in a combination of technology-led and application-led research". What is interesting is the discussion about whether a demonstrator or "proof of concept" is relevant:
"The problem, very often, is that there is no actual concept to be proven. Either the concept has already been proven viable (there really is no need to prove again that we can build a context-aware tour guide), is never in any doubt (we know we can build location-based services) or is not actually proved by the demonstrator (Nigel Davis)"
Some excerpts I found interesting for my research practice, they are rant-oriented but quite true:
"Attendees generally felt that too many ubiquitous computing projects focus on applications addressing trivial problems (turning lights on and off remotely, finding others with similar interests at conferences, and so on) (...) ubiquitous computing researchers often enjoy “relatively problem-free lives.” So, we should be keen to look beyond our own experiences when choosing application domains. For example, what opportunities exist to address problems in war zones or refugee camps? (...) ubiquitous computing researchers often reimplement applications from scratch, rather than sharing code and building on each other’s work. regard much of this implementation work as research. (PlaceLab is a good exception) (...) Researchers commonly evaluate ubiquitous computing applications solely in the context of small lab-scale user studies. (...) applications are often evaluated only against themselves (for example, “our participants said that they found this application useful”"
Why do I blog this? it's always interesting to read or hear about this sort of discussion in research. I have to admit that I have encountered lots of the problems described here, and as a researcher I sometimes do these mistakes (for instance in CatchBob it would have been good to run a longitudinal evaluation and less a field experiment, but it's a matter of time...). Also, it seems that human beings have a good tendency to reinvent the wheel (e.g. recreating new systems that do the same as others). This does not mean that nothing can be done and the paper concludes with 4 relevant propositions: choose problems and applications carefully, share technical infrastructures, evaluate applications in realistic settings and perform comparative evaluations.