HandShake – Using Hand Motion Recognition to Enable Communication

Oppenheim, Matthew and McIntyre, Fil (2020) HandShake – Using Hand Motion Recognition to Enable Communication. Communication Matters, 34 (1). pp. 19-21. ISSN 0969-9554

Abstract

HandShake is designed to enable people who cannot use physical controllers such as buttons or joysticks, but who can make an intentional hand or arm movement, to be able to interact with switchable Augmentative and Alternative Communication (AAC) software. Videos and links to the project software and documentation can be found on the project website [Oppenheim]. Beaumont College in Lancaster educates around 100 students with a spectrum of disability, mostly resulting from cerebral palsy. Many of these students require AAC. The Technologists at Beaumont College identified a group of students requiring AAC who can perform intentional hand movements but who are unable to interact with the physical controllers required to operate AAC. This group find operating physical controls difficult due to having a limited range of hand motion or being unable to accurately control their movements. Some members of this group have hand motions that are too violent and uncontrolled to safely use physical controls without a risk of hurting themselves. HandShake is designed to enable this group to interact with switchable AAC. The acceleration that the student’s hand is moving with is constantly monitored using a small circuit board worn on the wrist. When the acceleration exceeds a preset limit, a key press command is sent to the student’s AAC software. The threshold of acceleration at which the system triggers can be changed without having to touch the device on the wrist. This allows the system to be adjusted to cater for both gentle and energetic motion characteristics. HandShake uses the BBC micro:bit [BBC] educational circuit board as the accelerometer sensor platform. As this board is deemed safe to give to 11-12 year-olds in British schools, there are no safety concerns about using the device with the students at Beaumont. This allows the project to be safely replicated using cheap and easily-available hardware. Significant prior work exists for unique hand gesture recognition for people with no impairment to their hand motion [Rautaray & Agrawal 2015]. No prior research on using hand gestures for participants with impaired mobility as assistive technology was found.