Research

Measuring Situation Awareness in a Collaborative Game

in

It seems that ThoughtLink is working on a very similar project of ours. Together with the Center for Naval Analyses (CNA), they conducted an experiment to measured the effect of different modes of communication and visualization on a distributed team's shared situational awareness. This project was done for the Defense Advanced Research Projects Agency (DARPA) in support of the Wargaming the Asymmetric Environment (WAE) program. They investigated collaborative situation with the following factores: team members had to share information to do well, their decisions could be directly and easily recorded, the measure of their decisions would describe the degree of shared situational awareness of the team. Situation Awareness is according to Endsley (1995):

the perception of the elements in the environment within a volume of space and time, the comprehension of their meaning, the projection of their status into the near future, and the prediction of how various actions will affect the fulfillment of one's goals.

The game they used is ScudHunt:

SCUDHunt is a simple, short, abstract game of command, control, communications, intelligence, surveillance and reconnaissance (C3ISR) played by a team of 4 to 7 players. The game requires group decision-making and allocation of scarce resources under conditions of time pressure and uncertainty.

The goal of the game is for team members to correctly determine (within a specified timeframe) where three Scud launchers are located, on a 5 by 5 grid. Each team member controls a different sensor, and team members must share their sensor results in order to identify the launcher locations. In each turn, team members decide, typically in a collaborative process, where to locate each sensor. The results for a sensor are returned to the team member controlling it. Results include: X - launcher found; O - no launcher in the square; ? - not sure. Some sensors can also be killed or temporarily disabled on a turn.

At the end of each turn, based on the search results to date, each team member nominates at least three grid squares in which the launchers might be located. The overlap among the nominations reflects the team's shared situational awareness. If there is no overlap, every team member will vote for a different set of grid squares. If there is complete shared situational awareness, each member will vote for the same set of grid squares.

We measured SSA as: the total number of nominated squares divided by the number of unique grid squares nominated. Team SSA scores vary from 1 to the number of team members. As an example, given 4 team members each of whom nominates three squares, if there is minimal SSA, each member will nominate different sets of squares. Then the total number of nominated squares is 12 (4 members x 3 squares/member) and the number of unique grid squares is also 12 (there is no overlap), and the team's SSA score is 1. If the same team has complete SSA and each member votes for the same set of grid squares, then their team score will be 4: 12 squares nominated overall divided by 3 unique nominations.

There is a full report about this: Gaming and Shared Situation Awareness. Why do I blog this? in the European project about 'mutual modeling' we are working on, there is also this idea of using a collaborative game as a paltform to extract information concerning collaborative behavior. My interest towards this game is twofold:

  • to know how researchers benefit from using a game as a platform to extract data, eliciting collaborative behavior
  • to see which results they obtained about the use of communication tools and their impacts on situational awareness

CatchBob! users maps

in

The picture below describes the path did by 10 groups of CatchBob!, 5 played with the location-awareness tool, 5 played without. In this case, the object was located on the bridge between building CE and CM (which actually plays the role of a Schelling Point somehow since it's a bottleneck in the campus). Why do I blog this? I am trying to figure out how to visualize groups/players exploration of the campus. This kind of representation allows to see which are the differences among individuals and groups when exploring the field.

New Receiver issue

in

The new issue of Vodafone's magazine Receiver. It's about digital mobilization the East Asian way. There are great papers by well-known authors as Daisuke Okabe and Mizuko Ito, Antony Townsend or Atau Tanaka. I am always impressed by the quality of this content. Paper I found very relevant for my research:

Why do I blog this? unlike lots of carriers magazines, receiver is very valuable, it offers great insights by relevant researchers in the world.

Latency and user-behavior in video games

in ,

Latency and User Behaviour on a Multiplayer Game Server by Tristan Henderson - Networked Group Communication : Third International COST264 Workshop, NGC 2001, London, UK, November 7-9, 2001. Proceedings Editors: J. Crowcroft, M. Hofmann (Eds.): Chapter: p. 1.

Multiplayer online games represent one of the most popular forms of networked group communication on the Internet today. We have been running a server for a first-person shooter game, Half-Life. In this paper we analyse some of the delay characteristics of different players on the server and present some interim results. We find that whilst network delay has some effect on players' behaviour, this is outweighed by application-level or exogenous effects. Players seem to be remarkably tolerant of network conditions, and absolute delay bounds appear to be less important than the relative delay between players.

Why do I blog this? latency, uncertainties and discrepancies are important question when users interact with applications. This paper is a good introduction to how this might influence players' experience and how delays can impact the players' behavior. That is an issue which interest us at the lab; we would like to know how uncertainties and latency could modify the way mobile users of CatchBob! interact with the game and among the group. Actually this is more related to Fabien next-to-be research project :) Other related references about this topic:

Roadmap for current european project (summertime)

in

Roadmap for the MutualModeling project:

  1. State of the art document for end of september: literature review / paradigm / methodology to grasp Mutual Modeling / main results / perspectives with regard to our project. I have to take care of “shared cognitive environment" (Sperber and Wilson), “grounding” (Clark), “coordination theory” (Malone and Crowston)
  2. Explore the methods we discussed for September. First, sketch the experiment design + concrete needs
  3. Think about workshop or seminar: a list of potential speakers
  4. Conference/journals list
  5. Meeting “mutual modeling acts” in july

Conference paper about Catchbob! / Article de conférence sur CatchBob!

in

I wrote a paper for the french Human Computer Interaction Conference that has been accepted.

Empirical Study of Geolocation Use in Mobile Collaboration by Nicolas Nova, Fabien Girardin and Pierre Dillenbourg

This paper describes a collaborative location based application in built-in environment. This system is basically a platform that aims at exploring the impacts of positioning technologies on collaborative processes involving geographically dispersed teammates. The paper presents the platform as well as the first results of an ongoing study in which we tested two interfaces: one with a location awareness tool and another without. Results show that being given the position of the partners does not lead to better performance. Moreover, participants who were not provided with the position of others sent more messages to each other’s and made more mistakes when we asked them to draw the path taken by their partners.

J'ai écrit un article pour la conférence Interaction Homme Machine qui a été accepté

Etude empirique de l’utilisation de la géolocalisation en collaboration mobile par Nicolas Nova, Fabien Girardin et Pierre Dillenbourg

Cet article décrit une application de géolocalisation de groupe en environnement construit. Ce système constitue une plateforme d’exploration de l’impact des technologies de positionnement spatial sur les processus de collaboration entre partenaires géographiquement dispersés. L’article présente la plateforme ainsi que les premiers résultats d’une étude testant deux interfaces : l’une affichant avec le positionnement des partenaires et l’autre sans. Les résultats ne montrent pas de modification au niveau de la performance. Par contre les participants qui n’avaient pas ce positionnement com-muniquent plus et font beaucoup moins d’erreurs quand on leur demande de se remémorer le chemin parcouru par les partenaires.

Starbucks, tv, microwave ovens and distributed cognition

in

A talk by David Kirsh at a conference in lyon, France: Cost Structure and the Design of Environments : Design Lessons from Starbucks, watching TV and Microwave Ovens:

I present a micro-analysis of espresso making at Starbucks, watching TV and using microwave ovens to show difficulties with the cost function approach to design. An intuitive idea, discussed originally by Herbert Simon but pursued by many, is that the artifacts, technology, and resources in a setting define a cost structure for the activities performed there. This cost structure is the result of applying a cost function to each task-relevant action at each task-relevant state that can occur in that setting. The role of technology and design is to reshape the setting in a way that restructures the costs and benefits of actions so that more efficient routines emerge. The result is that tasks can be performed more easily and quickly.

I show why this model is seductive by looking at how the activity of watching TV has changed over time. I then look at the realities of activity in Starbucks to show that there are more factors determining effective routines, and hence good design, than those mentioned in cost functions. I also look at the remarkable design world of microwave ovens and related kitchen appliances to show that the dynamics shaping design are more complicated than standard efforts at formalizing accommodate. We need additional concepts to understand the dynamics of design

One can also find here what Kish values in Starbucks cups:

Thus, the marks on the paper cups used in Starbucks coffee houses, carry a special meaning to the staff. They are a code that has to be learned. Such marks are worth studying because they are an instance of a trick which, at one time, only the most experienced coffee making staff practiced. Yet now they have been institutionalized in workflow so that in rush hour all cashiers specify drinks in an unambiguous code that compensates for interruptions, noise and memory limitations on the production side. To understand how annotations help workflow we need to undertake a variety of careful studies in everyday venues.

Why do I blog this? because I like Kirsh's approach to distributed cognition.

Research paper idea

in

Idea for a research paper: how location-awareness has been interpreted and supported in different multi-user environments (textual virtual reality, 2d games, 3d games, mobile games) as technologies have advanced? I already have material for this, especially about MOO games, 3d games and a mobile game. This could be used to exemplify how location-awareness is employed by users.

CAIF Workshop Notes

in

People interested in reading what's happening at our workshop may have a look at our collaborative notes!So far, it's funny to see every speaker clearly defining his/her role ("I am a computer scientist and not an architect or a psychologist", "We are architects and not...", ...), of course everybody feels like being part of Human Computer Interaction but it's a so fuzzy notion that people seems to set the boundaries.

Location-Aware systems evaluation

in ,

Thank you Philip for pointing me on this relevant research paper: Deploying and evaluating a location-aware system (by R. K. Harle and A. Hopper from University of Cambridge, UK), International Conference On Mobile Systems, Applications And Services,Proceedings of the 3rd international conference on Mobile systems, applications, and services (2005)

Location-aware systems are typically deployed on a small scale and evaluated technically, in terms of absolute errors. In this paper, the authors present their experience of deploying an indoor location system (the Bat system) over a larger area and running it for a period exceeding two years.A number of technical considerations are highlighted: a need to consider aesthetics throughout deployment, the disadvantages of specialising sensors for location only, the need for autonomous maintenance of the computational world model, the dangers in coinciding physical and symbolic boundaries, the need to design for space usage rather than space and the need to incorporate feed-back mechanisms and power management. An evaluation of long term user experiences is presented, derived from a survey, logged usage data, and empirical observations. Statistically, it is found that 35% wear their Bat daily, 35% characterise their Bat as useful, privacy concerns are rare for almost 90% of users, and users cite the introduction of more applications and the adoption of the system by other users as their chief incentives to be tracked.Thia paper aims to highlight the need to evaluate large-scale deployments of such systems both technically and through user studies.

Why do I blog this? another reference in my literature review about evaluating the use of location-aware technologies! There is now a certain amount of paper about it, I should put all of this into a more formal paper with the emerging patterns. What I like in this reference is the "ong term user experiences" evaluation as well the use of mix data: surveys, logged usage data, and empirical observations.

In-Use, In-Situ: Extending Field Research Methods

in

Relevant conference to present the methodology we used to study CatchBob: In-Use, In-Situ: Extending Field Research Methods (27-28 October 2005.) at BCS London, The Davidson Building, 5 Southampton Street, Convent Garden, London WC2E 7HA. Organised by the Interaction Design Centre, Middlesex University, and the BCS HCI Education & Practice SubGroup, BCS HCI Group.

The workshop will explore the state of art in field research, experimental work, and other methods of data collection relevant to designing/evaluating technology ‘in-use, in-situ’. An aim is to identify gaps and problems for the future development of design methodology. Researchers from domains spanning the social and computing sciences, engineering, design, humanities, sciences, will have the opportunity to both share their experiences and reflect on the fundamentals of complex socio-technical systems and human-centered technology. The first day of this 2-day event will consist of a tutorial on‘Activity theory in the “fields” (by Seth Chaiklin). (...) The second day of the workshop will be a forum for discussion and presentation. We invite the submission of position papers on the theme of ‘in-use, in-situ’ evaluation. Papers should be no more than 4 pages in length and should be submitted by 22 July 2005.; Position papers will be reviewed by a programme committee, and selected authors will be invited to present their work on the second day of the event. Research areas include, but are not limited to the following:

* Ethnographic or similar studies of technology in use * Evaluation and validation of technology * Assessing the impact of technological interventions on work and other systems * Contextual and collaborative approaches to design * Studies of cooperative work * Tailoring and appropriation of technology * Novel technologies and contexts of interaction

Submissions should be sent to: Paola Amaldi (p.amaldi-trillo@mdx.ac.uk) by 22 July 2005. Authors will be notified of acceptance by 2 September 2005.

Why do I blog this? We may send something.

How Mogi-Mogi has been developed

in ,
If you happen to read french, there is a must-read today: Les NTIC comme architectures de la rencontre pour une société d'individus: Le cas du développement d'un jeu de rôle mobile-internet basé sur la géo-localisation des terminaux (ICTs and the engineering of encounters: A case study of the development of a mobile game based on the geolocation of terminals) by Christian Licoppe et Romain Guillot (France Telecom R&D).

It's actually a study carried out by researchers at France Telecom about how french company Newt Games imagined its next game, a mobile multi-user role palying game, which has to be scaled down into something simpler: Mogi Mogi (more information about the game in Feature).

Anchored into sociological (as well as socio-cognitive theories � la Kirsh, Norman, Suchman), the paper explores how a game at first targeted to the specific mobile game fans turned out to be accessible to everybody (ranging from simple users to hardcore gamers) and that can support encountering/matchmaking. The most interesting part (with regard to my interest, no offense for the writers) is simply the description of the game development process. Of course, in this paper, the name of the developers/company is not cited, nor the name of the game but it's easy to guess that it's Mogi Mogi (based on various cues: a snapshot!!!!, partnership with KDDI, link with Innovacell).

The very interesting point in this paper is how the mobile game developer reshuffled their project due to various constraints (technology, market, financial pressure, management, critics by game editor Ubi Soft).

There is actually an english version of this paper, presented at Mobile Leisure and the Technological Mediascape (Manchester, 2005): ICTs and the engineering of encounters: A case study of the development of a mobile game based on the geolocation of terminals:

By means of tests and user feedback, designers initially oriented towards the concept of a multi-player role playing game for mobile phones, targeted towards a specific audience, will shift their design strategy. They will gradually grasp the potential represented by the possibility of users “seeing” their mutual positions on mobile screens in order to enter into contact with one another. Their design work will focus on the engineering of encounters, through an innovative geolocalised service which is now oriented towards any mobile phone user (and not only gamers) – a generic device that anyone could use in principle. The design trajectory moves away from the development of a highly scripted, distinctive game towards the development of a generic information and communication technology.

Since the services they design are based on location tracking, they are particularly interesting from a sociological standpoint. Geolocation embeds issues of space and place directly into the engineering of mediated encounters. Up to now, electronic encounters were a characteristic feature of Internet world, i.e. in situations where actors use a connected personal computer. The development of mobile technology actually introduces original possibilities of exploiting cell phone tracking (in wireless network or through satellite positioning) to engineer disembodied meetings “on screens”. Since mobile phones almost always accompany their owners as they move about, a geographic position (that of their “geolocated” terminal) can be associated with personal electronic identities. The mobile phone screen may become a map of the cityscape, and icons or avatars represent the location of the players that move in it.

Mobile game and the use of replay tools to analyse

in

Recording and Understanding Mobile People and Mobile Technology by Paul Tennent and Matthew Chalmers, in proceedings of the first international conference on e-social science, Manchester, UK.

We present an approach to recording and understanding the activity of people moving and interacting with each other via technologies such as mobile phones and handheld computers. Our focus is the combination of observational techniques, usually based on video recordings, and system–based techniques that log or instrument the technologies in use. At a higher level, we explore tools to allow sociologists and computer scientists to interact around a coherent visualisation that coupled resources usually associated with just one of these two communities of research practice. The Replayer system supports the creation of system logs, and the visualisation of the results in a display that is synchronised with video and audio recordings. We present a case study showing how Replayer was used in the evaluation of a mobile multi-user system called Treasure, highlighting evaluation results that would not easily have been discovered by more traditional means.

Why do I blog this? because with CatchBob! we are also working on this issue; we have a replay tool that we use to analyse the data extracted from the game and confront it to the players. They then have a trace of the activity on which they can reflect to explain us what happened. Paul Tennent has an interesting webpage in which he describes more this topic, which is his PhD resreach issue. I like his conclusion:

Location-aware mobile systems are by their very nature difficult to evaluate. Combining logs from multiple sources to create a coherent view of a system state, can prove to be of great value when performing post-hoc analysis of a system however this usually requires considerable effort and often custom built tools. This system aims to simplify this process by providing a generic toolset for the creation and visualisation of log data, applicable across a wide variety of applications.

New Scientist about Location-Based Games

in , ,

Science E-Zine New Scientist has a good piece of location-based games: Gamers turn cities into a battleground. The article summarizes the best-known project such as "Uncle Roy..." or "Pac-Manhattan", "Digital Street Games" and the game designed by It's Alive. It also describes next avenues. Some exerpts:

While many of the first real-world games involved using separate GPS receivers and handheld computers, mobile phones and PDAs that integrate such technology are catching up. "There's an evolution using the mobility of the phone to create completely new gaming experiences," says Tom Söderlund, who worked as a games producer for Swedish games company It's Alive, based in Stockholm. "I think we are going to see more and more games that blend with our real lives."

Definitely, instead of transposing console games to mobile devices, the crux point here is to take advantage of the handheld features (mobility, voice, positioning...) to design innovative game scenarios.

The end of the paper is the msot interesting since it describes what's next, focusing on Gizmondo's plan:

Games console makers are also embracing the trend. Portable console maker Gizmondo is soon to launch Colors, a gangland game where players play a conventional arcade game to earn credits and money. These are then used to buy turf in the real world - Soho in London, say. Walk into a Soho cafe and attempt to play Colors, and the GPS embedded in the console might tell you you're playing on another gang's patch, and you need to beat them in a virtual fight to claim the turf and continue.

The company has even bigger plans, developing a game that exploits a digital camera already built into the console. Virtual creatures live at specific GPS coordinates, and when a player views the location through the camera they will see the real world with a three-dimensional animated digital creature laid over the scene.

Finally, it points the many problems that arise with these location-based games:

For some games to work, you need a quorum of players. "If I'm a good terminator trying to find a bad terminator to fight, and the only bad one lives in Sweden, then I'm not going to see much action,"

And in Uncle Roy, for example, not only does the game involve innocent bystanders - the woman dressed in black who Matt followed had no knowledge she was taking part in the exercise - but it culminates in the street player climbing into a stranger's car, which means the player has to trust the organisers.

game designers face the challenge of how to preclude "cyber-stalking", and protect the safety of the public and players, especially children, who might wander into unsafe situations or places.

A very interesting summary of what's going on lately!

Our location based application research

in ,
We just finished a document that shortly summarizes our research projects about location-based services at Craft. It can be downloaded here (pdf, 909kb).

The 4-pages document gives an overview of CatchBob!, Shoutspace and Stamps. It also presents few results about CatchBob! as well as the issue we have to dealt with while designing those applications.

It's not intended to be read by academics; we just did it as a teaser that explains to non-scholars persons interested by our projects.

Review of usability evaluation techniques

in

Ivory M., Hearst M. (2001) The State of the Art in Automating Usability Evaluation of User Interfaces. ,ACM Computing Services Vol 33(4), pp.470-516

Usability evaluation is an increasingly important part of the user interface design process. However, usability evaluation can be expensive in terms of time and human resources, and automation is therefore a promising way to augment existing approaches. This article presents an extensive survey of usability evaluation methods, organized according to a new taxonomy that emphasizes the role of automation. The survey analyzes existing techniques, identifies which aspects of usability evaluation automation are likely to be of use in future research, and suggests new ways to expand existing approaches to better support usability evaluation.

Why do I blog this? to complete my review of automatic user experience analysis. This paper provides a relevant summary all the current techniques. Even though it's mostly related to usability (which is something a bit too low-level for my ressearch needs), it gives a nice overview of the existing techniques. It's a good add-on to Mike Kuniavky's 'User Experience' book (which is one of the best introduction and how-to for user-experience research).

Location matters in collaborative problem solving

in

Random notes about my research project: I like this this "location matters" motto. The point is that even though IT modified how people carry out social activities by removing the notion of distance, space and location are still important features in terms of collaborative problem solving. "Location matters" relies on the hypothesis that people use location information in collaborative problem solving: knowing where the parters are can improve collaboration in terms of:

  • division of labor among the group (A goes there on level2, B goes here on level1 for instance)
  • doing inferences about others activities (past: where you have been can attest what you done, present: if you're at a specific point you may do something, future: heading in that direction can mean that you will do sth specific)
  • doing inferences about others' availability (A is in X then he may be busy)
  • building a shared understanding of the situation: having a map of the environment with potential actors' positions
  • re-shape communication and help referential commmunication: being close to an object you're talking about helps the others understanding that you're making reference to it

All of this relies on a "belief system" in which actors may/could assign meaning to people's actions based on location information. I would like to have a model of this (a probabilistic model?).

(thank you Hillevi for making think about it, now I have a clearer description of what I thought)

Phd report

in

Stuff to dig for the phd:

  • move forward with catchbob! try another few experiments with other interfaces (asynchronous awareness tool, self-declared positioning, no position at all)
  • from those experiments, pick up 2-3 variables to have a more precise (less exploratory) experiment; it's better if we have a more complex task, higher problem solving and more cognitively demand
  • replicating our experiments in this more complexe environment is a good way to get ride of classical critiques like "your results are bound to a specific task/environment".
  • computer model fro catchbob! try to investigate the old AI "belief system": it provides an interesting mathematical formalism to describe beliefs (like "I believe that my CatchBob partner is at that location" with a probability of 0.9). It may be of interest to model relation between perceived space and physical/real space