Aims

Understanding, (automatically) analysing and modelling people's affective and social behaviour is very important for multiple domains such as enhancing human-computer and human-agent interaction, improving gaming technology and players’ experience, behavioural analytics for the banking sector, etc. Accordingly, the aim of this module is to impart knowledge and ability needed to make informed choices of models, data, and techniques for sensing, recognition, and expression of human affective and social behaviour (e.g., smile, frown, head nodding/shaking, agreement/disagreement), and its use in the design of innovative interactive technology (e.g., interaction with virtual agents, robots, and games; single and multi-user smart environments, e.g., in a car; implicit (multimedia) tagging; clinical and biomedical studies, e.g., autism, depression, pain).

Syllabus

The following list provides a representative list of topics:

• Definitions, Overview and Applications: entertainment technology/gaming/arts; clinical and biomedical studies, e.g., autism, depression, pain; etc.; implicit (multimedia) tagging; affective wearables);
• Emotion theories: psychology, cognitive science and neuroscience;
• Emotional design: frameworks (e.g., visceral, behavioural, reflective design)
• Sensing emotions: vision, audio, bio signals, text; data acquisition and annotation, tools and databases;
• Processing: extracting meaningful information and features from multiple modalities;
• Recognition and Evaluation: applying machine learning techniques;
• Affect and expression synthesis and generation (virtual characters, robots, etc.);
• Hands-on programming work with relevant libraries relevant for affective and behavioural computing;
• Evaluation: automatic analysers, and emotionally and socially intelligent systems;
• Ethical issues and open challenges

Objectives

The aim of this module is to impart knowledge and ability needed to make informed choices of models, data, and techniques for sensing, recognition, and expression of human affective and social behaviour (e.g., smile, frown, head nodding/shaking, joy/depression), and its use in the design of innovative interactive technology (e.g., interaction with virtual agents, robots, and games; single and multi-user smart environments, e.g., in a car; implicit (multimedia) tagging; clinical and biomedical studies, e.g., autism, depression, pain).

On completion of this module, students should:

• Understand the challenges in human-human affective and communicative interaction (e.g. not what is said but how it is said – using the body, head, face, intonation, etc.) and its implication to Human-Computer Interaction;
• Demonstrate knowledge in current theories and trends in designing emotionally and socially sensitive interactive technology, as well as recent advances in human audio/visual/bio signal processing, and recognition using machine learning and pattern recognition techniques;
• Comprehend and apply (appropriate) methods for collection, analysis, representation and evaluation of human affective and communicative behaviour data;
• Demonstrate ability to computationally analyse, recognise and evaluate human affective and social behaviour;
• Enhance programming skills for human affect and behaviour analysis and understanding;
• Demonstrate critical thinking, analysis and synthesis while making a decision on 'when' and 'how' to incorporate emotions and social signals in a specific application context, and gain practical experience in proposing and justifying computational solution(s) of suitable nature and scope.

Assessment

Practicals: 25%
Seminars: 25%
Project report: 50% (written report)

Recommended reading

Picard, R. (2000). Affective Computing. MIT Press.
Gokcay, D. & Yildirim, G. (Eds)(2011). Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives.. IGI Global, http://www.igi-global.com/book/affective-computing-interaction/41787.
Salah, A. A. & Gevers, T. (Eds)(2011). Computer Analysis of Human Behavior.. 2011, XV, 409 p. http://www.springer.com/computer/hci/book/978-0-85729-993-2.
Petta, P., Pelachaud, C. & Cowie, R. (Eds) (2011). Emotion-Oriented Systems: The Humaine Handbook.. Springer Verlag. Amazon link
Journals:
1) IEEE Transactions on Affective Computing http://www.computer.org/portal/web/tac/about
2) IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=3477
3) International Journal of Synthetic Emotions http://www.igi-global.com/journal/international-journal-synthetic-emotions-ijse/1144
Conference proceedings:
1) ACII: Affective Computing and Intelligent Interaction
2) ICMI: ACM International Conference on Multimodal Interaction
3) FGR: IEEE Conference on Automatic Face and Gesture Recognition
Online Resources:
1) The HUMAINE Portal: http://emotion-research.net/
2) The SSPNet Portal: http://sspnet.eu/