Can you read minds? The answer is most likely ‘yes’. You may not consider it mind reading but our ability to understand what people are thinking and feeling from their facial expressions and gestures is just that. People express their mental states all the time through facial expressions, vocal nuances and gestures. We have built this ability into computers to make them emotionally aware.

The ability to attribute mental states to others from their behaviour and then to use that information to guide our own actions or predict those of others is known as the ‘theory of mind’. Although research on this theory has been around since the 1970s, it has recently gained attention due to the growing number of people with Autism conditions, who are thought to be ‘mind-blind’. That is, they have difficulty interpreting others' emotions and feelings from facial expressions and other non-verbal cues.

Our computer system is based on the latest research in the theory of mind by Professor Simon Baron-Cohen, Director of the Autism Research Centre at Cambridge. His research provides a taxonomy of facial expressions and the emotions they represent. In 2004, his group published the Mind Reading DVD, an interactive computer-based guide to reading emotions from the face and voice. The DVD contains videos of people showing 412 different mental states. We have developed computer programs that can read facial expressions using machine vision, and then infer emotions using probabilistic machine learning trained by examples from the DVD.

Machine vision is getting machines to ‘see’, giving them the ability to extract, analyze and make sense of information from images or video, in this case footage of facial expressions. Probabilistic machine learning describes the mechanism of enabling a machine to learn an association between features of an image such as facial expression and other classes of information, in this case emotions, from training examples. The most likely interpretation of the facial expressions is then computed using probability theory.

(Description by Sallie Robins for the Royal Society Summer Science Exhibition.)

Applications

Machine versus people testing of our system has shown the computer to be as accurate as the top 6% of people. But would we want computers that can react to our emotions? Such systems do raise ethical issues: Imagine a computer that could pick the right ‘emotional’ moment to try to sell you something. There are, however, applications with clear benefits including an emotional hearing aid to assist people with autism, usability testing for software, feedback for on-line teaching, and informing the animation of cartoon figures.

We have been working since 2004 on a wearable system that helps people with Autism Spectrum Conditions and Asperger Syndrome, with emotional-social understanding and mind-reading functions. Rana el Kaliouby, who was awarded a PhD for her work on the project, is currently implementing the first prototype of the system at the Massachusetts Institute of Technology’s Media Lab.

Metin Sezgin, who recently completed a PhD at MIT, has joined the team in Cambridge to look at ways of improving the inference of mental states by combining multiple sources of information, including biometric sensors. Tal Sobol-Shikler investigated the effects of emotions on non-verbal cues in speech for her PhD at Cambridge, and is now pursuing the work at Ben-Gurion University. Daniel Bernhardt, another research student, is extending the system to recognise further channels of affective communication such as posture and gesture. We are working with UCL to use this information in the animation of avatars. Shazia Afzal is looking at applications of affective inference to support on-line teaching systems.

Another important area is discerning drivers' mental states. If a driver gets lost while trying to find a route through an unfamiliar city in heavy traffic, the burden of understanding advice from a navigational system could actually be more of a hindrance than a help. Ian Davies is working with a major motor manufacturer on systems to detect when a driver is confused, distracted, drowsy or even upset, and adapt the car's telematic systems accordingly.

Further information

• Peter Robinson
• Mind-reading machines
• Video of the Royal Society Summer Science exhibit (needs RealPlayer)
• Mind-reading examples
• Empathic avatars
  Interactive control of music using emotional body expressions (video shown at CHI in 2008)
• Frequently asked questions

Selected publications

• Real-time inference of complex mental states from facial expressions and head gestures
  A book chapter giving technical details of the mind-reading system
• Generalization of a vision-based computational model of mind-reading
  A conference paper describing tests with members of the public
• Detecting affect from non-stylised body motions
  A conference paper discussing emotional inference from posture and gesture
• Affective video data collection using an automobile simulator
  A conference paper discussing an application to monitoring car drivers
• The emotional hearing aid - an assistive tool for autism
  A conference paper describing another possible application

Press coverage

The project attracted a lot of attention at the Royal Society's Summer Science Exhibitions in July and September 2006. It was covered by four television channels, a dozen radio stations, and a variety of other media. Google tracked over 100 on-line reports across all continents including the following.

• BBC report on monitoring car drivers, 5 March 2008.
• Leading Edge, BBC Radio 4, 6 July 2006.
• Hindustan Times, 28 June 2006.
• Times online, 26 June 2006.
• BBC online, 26 June 2006.
• Daily Telegraph, 26 June 2006.
• Reuters, 26 June 2006.
• New Scientist, 1 April 2006.
• BBC online, 29 March 2006.