Abstract

Mobile phones represent an ideal computing platform to monitor behavior and movement, since they are part of the everyday life of billions of people. One possible use of these technologies is arguably the support to sociology experiments which involve studying people's daily life and interactions. In the past, this analysis has been performed with the help of cameras (in home/working environments or in laboratories), by using voice recorders attached to people, and self reports using daily diaries or PDAs. However, these techniques may lead to biased results since people are aware of being constantly monitored. Instead, mobile phones offer an unobtrusive means of obtaining information about the behavior of individuals and their interactions.

EmotionSense is a framework for collecting data in human interaction studies based on mobile phones. EmotionSense gathers participants' emotions as well as proximity and patterns of conversation by processing the outputs from the sensors of off-the-shelf smartphones. This can be used to understand the correlation and the impact of interactions and activities on the emotions and behavior of individuals. In terms of system design, the key characteristics of this framework areprogrammability (social scientists can describe the sensing tasks using a declarative language), and run-time adaptation (social scientists can write rules to activate and deactivate sensors according to the user context).

Technical Aims

• Accurate emotion and speaker recognition based on off-the-shelf smartphones so that inexpensive large-scale deployments can be possible.
• Build efficient inference algorithms to extract high-level information from the available raw data of not always accurate sensors embedded in mobile phones.
• Build an efficient system for this class of resource-constrained devices (especially in terms of power consumption).
• The system should be easily programmable and customizable for different types of experiments with changing requirements.

Progress

We implemented the EmotionSense system on a Nokia 6210 Navigator phone using PyS60 for most of the components. The speaker and emotion recognition components are implemented in C++ since they are based on tools of the Hidden markov model ToolKit (HTK) suite for speech processing originally written in that language.

We conducted a social psychology experiment to evaluate the usefulness of the EmotionSense system for social scientists. The data extracted by means of the EmotionSense system running on the mobile phones were compared to information provided by participants by means of traditional questionnaires.The experiment was conducted for a duration of 10 days involving 18 users. This study showed that the information collected by EmotionSense can be used by social scientists in order to understand the patterns of interaction and the correlation of emotions with places, groups, and activity.

People

Project Collaborators

Publications

Awards

Media Coverage