Vibrotactile Feedback in Piano and Violin Playing: Actuation, Interaction, and Sensation

Abstract

The musician-musical instrument interaction can be regarded as a control loop where the musician adjusts his or her playing until the desired output of the instrument is achieved. When monitoring the instrument's behavior, different sensations are involved. The most important feedback for the musician is certainly the sound produced, but haptic feedback provides additional cues that are sensed at the interface of the instrument. Haptic feedback is the combination of force feedback - the mechanical reaction of the instrument - and vibrotactile feedback: the vibrations perceived through the contact points between the musician and the musical instrument. It has been shown that vibrotactile feedback contributes to the perceived quality of musical instruments. Furthermore, it has been suggested that vibrotactile feedback can support the precise control of finger forces. By assuming the control loop model of the musician-musical instrument interaction, a reaction of the musician to vibrotactile feedback can be expected. This thesis uses a new approach to investigate the role of vibrotactile feedback in piano and violin. Vibrations at the contact points between the musician and the musical instrument are explored by assessing the player's perceptions and measuring his or her reactions regarding the dynamics and timing of his or her musical performance. Additionally, vibration measurements at the contact points between the musician and the musical instrument are presented for pianos and violins. In an extensive set of measurements with a professional pianist, the piano key vibrations of eleven acoustic pianos are compared. It is demonstrated that vibration displacement time signals exceed the threshold of human vibration sensation for all evaluated instruments when notes are played on the lower half of the keyboard. We also show that the tonal part of the piano key vibrations displayed differences up to 20 dB among the evaluated instruments. To study the perception and reaction of pianists to the vibration levels in the keys, the vibrotactile feedback rendering system of an existing digital hybrid grand piano was extended. Four vibration levels were created and tested in an experiment with eleven pianists. By assessing the players' perception with ratings and free verbalizations, we found an optimum or ''sweet spot'' of vibration levels, which depends on the balance of the sound of the bass and treble piano keys. However, based on the evaluation of the dynamics and timing of the pianist's performances, we could not identify a general tendency that the pianists adapt their playing to the vibration levels and did not observe that vibrotactile feedback improves the control of finger forces. To study the role of vibrotactile feedback in the violinist-violin interaction, we amplified the vibrations at the contact points between the player and the instrument with actuators by reinforcing the inherent vibrations of an acoustic instrument. Hence, six vibration conditions were designed and evaluated in a study with a group of eight violinists. The players' perceptions of the instrument were evaluated regarding the instrument's playability and sound. Based on the players' ratings, some vibration conditions improved the judged sound and playability of the instrument. The players' reactions were evaluated via a finger force sensor integrated into the fingerboard of the violin. Furthermore, we demonstrated that the force exerted by the fingers of the left hand on the fingerboard had 3 to 11 % lower median values and lower variances for most of the vibration conditions relative to the non-actuated condition. The lower variance could indicate more precise or more constant control of the left hand finger forces during the performance. This adaption of the performance to the instrument's vibrotactile feedback is consistent with the players' self-assessment that they adapt their playing to changes in the perceived response of the instrument. For both instruments, we observed striking interaction effects between auditory and vibrotactile sensations. In addition to previously known effects, for example, changes in the perceived loudness and timbre, a sensation of reverb or ''body feeling'' in the sound was evoked for some of the feedback conditions in both instrument studies.