Stumbled across this workshop about Evaluation Methodologies for Ubiquitous Computing

Interactive systems, and in particular, ubiquitous computing pose more complex evaluation methodologies than non-interactive text retrieval. However, there are several possibilities. One possibility is to start by conducting evaluations on the various aspects that make up subsystems of ubiquitous computing: perceptual user interfaces, dynamic service discovery, wireless networking services, distributed data systems and input and output using distributed user interfaces. Issues here would be to establish metrics and evaluation methodologies for individual components and to determine what, if anything, successful evaluations imply about the entire system.

Another possibility would be to evaluate individual systems, end -to-end, as they are built using traditional usability evaluation methodologies. This would give us information about individual systems and would perhaps allow researchers more flexibility in choosing particular domains. Would this necessitate changes in usability evaluation methodologies? Typical usability metrics are effectiveness, efficiency, and satisfaction. Are these valid metrics for technology that is still under development? Is it feasible to compare systems across domains of use?

Another issue for evaluation of ubiquitous computing systems is that we would like to employ rapid evaluation methods whose results will be available in time to influence the final design. Creation of such methods and their validation would represent an important contribution to the HCI community.