As automation is becoming more prevalent across everything from military and healthcare settings to everyday household items, it is necessary to understand the nature of human interactions with these systems. One critically important element of these interactions is user trust, as it can predict automated systems’ safe and effective use. Past research has evaluated individuals’ trust in automation through a host of different assessment techniques such as self-report, physiological, and behavioral measures. However, to date, there has been little evaluation of the convergence across these measures in a real-world environment. Convergence across measures is a useful tool in understanding the mechanisms by which a cognitive construct is impacted and providing greater confidence that any single measure is evaluating what it purports to measure.
The present study used an autonomous golf cart that drove participants to different locations around the campus of James Madison University while a camera recorded them. In addition, participants were given the AICPR and TOAST to evaluate their complacency potential and trust, respectively. Researchers coded videos for verification/checking behaviors (i.e., participants looked at or interacted with the GUI used to control the cart) and nervous behaviors (i.e., bracing, fidgeting, etc.). Additionally, environmental ‘obstacles’ such as pedestrians, food-delivery robots, and construction were also coded for by watching a front-facing camera. Results indicate a disconnect between the self-report and behavioral measures evaluating trust. However, there was a relationship between the coded nervous behaviors and verification behaviors and a relationship between those and the presence of obstacles.
This lack of convergence across measures indicates a need for future research to understand whether this non-convergence represents shortcomings with the measures themselves, the existing definition of trust as a construct, or perhaps indicates that there is a nuance that can be afforded by some measures over another.