Predicting Situation Awareness from Physiological Signals

Situation awareness (SA)–comprising the ability to 1) perceive critical elements in the environment, 2) comprehend their meanings, and 3) project their future states–is critical for human operator performance. Due to the disruptive nature of gold-standard SA measures, researchers have sought physiological indicators to provide real-time information about SA. We extend prior work by using a multimodal suite of neurophysiological, psychophysiological, and behavioral signals, predicting all three levels of SA along a continuum, and predicting a comprehensive measure of SA in a complex multi-tasking simulation. We present a lab study in which 31 participants controlled an aircraft simulator task battery while wearing physiological sensors and responding to SA ‘freeze-probe’ assessments. We demonstrate the validity of task and assessment for measuring SA. Multimodal physiological models predict SA with greater predictive performance ($Q^2$ for levels 1-3 and total, respectively: 0.14, 0.00, 0.26, and 0.36) than models built with shuffled labels, demonstrating that multimodal physiological signals provide useful information in predicting all SA levels. Level 3 SA (projection) was best predicted, and level 2 SA comprehension) was the most challenging to predict. Ablation analysis and single sensor models found EEG and eye-tracking signals to be particularly useful to predictions of level 3 and total SA. A reduced sensor fusion model showed that predictive performance can be maintained with a subset of sensors. This first rigorous cross-validation assessment of predictive performance demonstrates the utility of multimodal physiological signals for inferring complex, holistic, objective measures of SA at all levels, non-disruptively, and along a continuum.