Demonstrating TapType for mobile ten-finger text entry anywhere

Abstract

We demonstrate a mobile text entry system that brings full-size ten-finger typing to everyday surfaces, allowing users to type anywhere. Our wearable wristband TapType integrates accelerometers that sense vibrations arising from finger taps against a passive surface, from which our Bayesian neural network estimates a probability distribution over the fingers of the hand. Given a pre-defined key-finger mapping, our text entry decoder fuses these predictions with the character priors of an n-gram language model to decode the input text entered by the user. TapType combines high portability with sustained rapid bimanual input across the full space, which we demonstrate at the example of supplementing text input on mobile touch devices, in eyes-free scenarios using audio feedback, and in a situated Mixed Reality scenario to enable typing outside visual control with passive haptic feedback.