Affective Computing

Aims

Computationally analysing and modelling people's socio-emotional behaviours is very important for multiple domains such as enhancing human-AI, human-agent and human-robot interactions; creating personalized learning environments, behavioural analytics for assessing and improving people’s comfort, healthcare and wellbeing; designing engaging and adaptive training environments and games, etc.

Accordingly, the aim of this module is to impart knowledge and ability needed to make informed choices of models, data, and machine learning techniques for sensing, recognition, and generation of affective and social behaviour (e.g., smile, frown, head nodding/shaking, agreement/disagreement), and its use in the design of innovative interactive technology (e.g., interaction with virtual agents, robots, and games; single and multi-user smart environments, e.g., in-car/ virtual / augmented reality, for public speaking and cognitive training; clinical and biomedical studies, e.g., autism, depression, pain) while addressing the ethical issues (e.g., privacy, bias) arising from the real-world deployment of these systems

Syllabus

The following list provides a representative list of topics:

• Introduction, definitions, and overview
• Emotion theories
• Sensing from multiple modalities (e.g., vision, audio, bio signals, text)
• Data acquisition and annotation
• Signal processing / feature extraction
• Automatic recognition / prediction and evaluation
• Behaviour synthesis / generation (e.g., for embodied agents / robots)
• Emotional design frameworks
• Advanced topics and ethical considerations (e.g., bias and fairness)
• Applications (via seminar presentations and discussions)
• Guest lectures (various topics - e.g., commercialising affective computing products)
• Hands-on programming work (i.e., mini-project)

Objectives

On completion of this module, students will:

• Understand the challenges in human-human affective and communicative interaction (e.g. not
  what is said but how it is said – using the body, head, face, intonation, etc.) and its implication to
  human-computer interaction;
• Demonstrate knowledge in current theories and trends in designing emotionally and socially
  sensitive interactive technology, as well as recent advances in human audio/visual/bio signal
  processing, and recognition using machine learning techniques;
• Comprehend and apply (appropriate) methods for collection, analysis, representation and
  evaluation of human affective and communicative behaviour data;
• Demonstrate ability to computationally analyse, recognise and evaluate human affective and
  social behaviour;
• Enhance programming skills for human affect and behaviour analysis and understanding;
• Demonstrate critical thinking, analysis and synthesis while making a decision on 'when' and
  'how' to incorporate emotions and social signals in a specific application context, and gain
  practical experience in proposing and justifying computational solution(s) of suitable nature and
  scope.

Assessment

Seminar presentation: 20%
Participating in Q&A and discussions: 10%
Mini-Project: 70% (proposal, mid-term written report, final written report, code and presentation)

Recommended reading

Picard, R. (2000). Affective Computing. MIT Press.

Jeon, M. (2017). Emotions and Affect in Human Factors and Human-Computer Interaction. Academic Press. https://www.elsevier.com/books/emotions-and-affect-in-human-factors-and-human-computer-interaction/jeon/978-0-12-801851-4

Calvo, R., D'Mello, S., Gratch, J. and Kappas, A. (2014) The Oxford Handbook of Affective Computing. Oxford University Press. https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199942237.001.0001/oxfordhb-9780199942237

Tian, L., Oviatt, S., Muszynski, M., Chamberlain, B. C., Healey, J. & Sano, A. (2022) Applied Affective Computing. Association for Computing Machinery, New York, United States.  https://dl.acm.org/doi/book/10.1145/3502398

Journals:

1. IEEE Transactions on Affective Computing https://www.computer.org/csdl/journal/ta

Conference proceedings:

1. ACII: Affective Computing and Intelligent Interaction https://dblp.org/db/conf/acii/index
2. ICMI: ACM International Conference on Multimodal Interaction https://dblp.org/db/conf/icmi/
3. FGR: IEEE Conference on Automatic Face and Gesture Recognition https://dblp.org/db/conf/fgr/