A Weakly Supervised Approach to Emotion-change Prediction and Improved Mood Inference

Whilst a majority of affective computing research focuses on inferring emotions, examining mood or understanding the $\mathit{\text{mood-emotion}}\mathit{\text{interplay}}$ has received significantly less attention. Building on prior work, we (a) deduce and incorporate emotion-change ($\mathrm{\Delta }$) information for inferring mood, without resorting to annotated labels, and (b) attempt mood prediction for long duration video clips, in alignment with the characterisation of mood. We generate the emotion-change ($\mathrm{\Delta }$) labels via metric learning from a pre-trained Siamese Network, and use these in addition to mood labels for mood classification. Experiments evaluating $\mathit{\text{unimodal}}$ (training only using mood labels) vs $\mathit{\text{multimodal}}$ (training using mood plus $\mathrm{\Delta }$ labels) models show that mood prediction benefits from the incorporation of emotion-change information, emphasising the importance of modelling the mood-emotion interplay for effective mood inference.