Algorithmic Gaze Measure Development for Application in Real-World Scenarios

Abstract

Due to its wide applicability in natural and dynamic real-world environments, mobile eye-tracking (MET) has become increasingly popular across numerous subject areas. However, the use of MET devices comes with some additional hurdles compared to screen-based eyetracking systems. More precisely, apart from a general decrease in recording accuracy in mobile systems, during eye movement data analysis, the dynamically changing environment impedes the computation of advanced contextual gaze measures. Due to the continuously changing size and shapes of task-relevant objects in the recorded images, the gaze-object mapping procedure is still a predominantly time-consuming and manual data labeling process. Consequently, researchers across all subject applications have opted to mainly evaluate basic, non-contextual gaze measures. The resulting findings have been valuable for the understanding of the operator’s gaze movements, but the lack of contextual information has limited in-depth and application-specific insights. In recent years, the emergence of effective and accurate machine learning (ML) based object detection algorithms have provided a promising solution to automating gaze-object mapping. This work investigates different concepts of how ML methods can be incorporated for the development of context-rich gaze measures using MET from dynamic real-world stimuli. Furthermore, it investigates the use of algorithmic approaches to conquer the limits of traditional gaze mapping procedures. Finally, this work aims to advance the understanding of the potential for eye movement measures as input features for complex ML applications. To achieve these aims, four studies (Studies I-IV) were conducted over the course of this thesis. Study I explored the use of the algorithmic k-mer subsequence approach, typically used in DNA pattern analysis, for the quantification of visual expertise and the measurement of distinct expertise-related gaze pattern sequences. A multi-trial study was conducted with 28 novice and 2 expert subjects, and their eye movements were recorded over the course of 8 successive trials of a simplified airplane toy assembly task. Area-ofinterest (AOI) gaze sequences were transformed to string representation und patterns were investigated using k-mer subsequences of sizes k = {1, 2, 3, 4}. The results of the k-mer analysis showed distinct expertisedependent gaze patterns, while basic fixation-based measures showed only weak expertise-related trends. Additionally, novices showed a significant increase in expert-like gaze patterns with increased on-task experience, indicating the usefulness of this approach for measuring expertise development. Study II investigates the development of an ML-based method for measuring and quantifying peripheral vision in challenging and dynamic recordings. In detail, a mask-reinforced convolutional neural network (RCNN) object detection model was used to express a subject’s use of peripheral vision using the object-gaze distance (OGD). The OGD measures the distance between the central point of foveal vision to an object-of-interest (OOIs) using the 2D Euclidean pixel distance. The analysis of 11 actual surgical spondylosis procedure recordings conducted by two expert surgeons revealed that utilizing OGD mapping offers several advantages over the traditional one-OOI-per-fixation mapping procedure. Specifically, OGD mapping significantly enhances the interpretability of the data, as it generates time-series format data for each OOI and has the potential to uncover novel gaze patterns. Study III introduced the visual attention index (VAI) for evaluating visual attention in multi-object tasks based on the temporal information of fixations and the spatial location of the OOIs within the subjects' areas of vision. The VAI was computed and evaluated for two MET datasets of manual handling trials and visualized using a radar graph representation. The study demonstrated that the VAI provides a valuable gaze measure that includes information on peripheral attention for computing visual attention in multi-object tasks and, hence, it’s suitability for task-specific visual attention analysis and comparison between trials and subjects. Study IV investigates the potential of advanced gaze measures for enhancing the performance of egocentric action recognition (EAR) models. In this study, we proposed a gaze-enhanced action recognition model, termed the peripheral vision-based hidden Markov model (PVHMM). The performance of the model was evaluated using 43 MET recordings acquired during a procedural medical device handling study. The classification performance of the PVHMM was assessed based on basic and advanced gaze measure inputs and compared to the purely image-based CNN classifier, the VGG-16. The results indicated that the context-rich gaze measure significantly outperformed previous state-ofthe- art methods, highlighting the immense potential of enhancing the performance of complex ML-models by including eye movement measures. In conclusion, the studies conducted in this thesis have shown that algorithmic approaches can be successfully integrated into the development of advanced context-rich gaze measures. It was shown that ML models can significantly enhance traditional gaze mapping procedures, thus facilitating a more complete depiction of an operator’s gaze behavior and visual attention. As a result, the quantification of visual expertise and peripheral vision can significantly increase the ability to assess an operator’s performance in real-world applications. In areas with safety-critical applications, such as medical surgery, the use of advanced eye movement measures has shown the potential to improve ML model performances and may lay the foundation for the development of more efficient and targeted training protocols, a deeper understanding of the causes of human errors and the subsequent development of assistive methods such as predictive support systems.