Gaze-Hand Alignment : Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality

Lystbæk, Mathias and Rosenberg, Peter and Pfeuffer, Ken and Grønbæk, Jens Emil and Gellersen, Hans (2022) Gaze-Hand Alignment : Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality. Proceedings of the ACM on Human-Computer Interaction, 6 (ETRA). 145:1-145:18.

Text (nocopyright-145-CameraReady_Gaze_Hand_Alignment_ETRA2022)
nocopyright_145_CameraReady_Gaze_Hand_Alignment_ETRA2022.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.
Download (6MB)

Abstract

Gaze and freehand gestures suit Augmented Reality as users can interact with objects at a distance without need for a separate input device. We propose Gaze-Hand Alignment as a novel multimodal selection principle, defined by concurrent use of both gaze and hand for pointing and alignment of their input on an object as selection trigger. Gaze naturally precedes manual action and is leveraged for pre-selection, and manual crossing of a pre-selected target completes the selection. We demonstrate the principle in two novel techniques, Gaze&Finger for input by direct alignment of hand and finger raised into the line of sight, and Gaze&Hand for input by indirect alignment of a cursor with relative hand movement. In a menu selection experiment, we evaluate the techniques in comparison with Gaze&Pinch and a hands-only baseline. The study showed the gaze-assisted techniques to outperform hands-only input, and gives insight into trade-offs in combining gaze with direct or indirect, and spatial or semantic freehand gestures.