Despite the rising number of people with hearing impairments, sign language is a dying skill in Hong Kong. A survey of people with disabilities and chronic illnesses released by the Census and Statistics Department in 2014 found that more than 155,000 people have hearing difficulties in Hong Kong, but that only 2.5 percent, or fewer than 4,000, know sign language.
At present, there is only one school for the deaf in Hong Kong that uses sign language.
In the long run, the problem has a deep psychological impact on deaf people, as this results in them becoming isolated and potentially having limited careers if social acceptance toward the language is low.
The only institution in Asia offering research and professional training in sign linguistics and deaf education – the Centre for Sign Linguistics and Deaf Studies of the linguistics and modern languages department at the Chinese University of Hong Kong – has launched SignTown, a sign language online learning game featuring AI-based sign language recognition to promote understanding of the skill.
In the web-based, real-time sign recognition game, users control an avatar who is a newcomer to a town in a fictional world where everyone communicates in sign language. The use of an end-to-end open-source platform for machine learning to train the sign language recognition model enables users to learn and express themselves in sign language and get feedback with just a computer and a webcam.
Gladys Tang, the center’s director, said the project is the first step in the center’s Shuwa Project, which aims to develop an automatic translation model that can recognize natural sign language conversations and translate them into spoken language.
The project is a joint effort. As the leading authority, CUHK provided the academic expertise for the development of the sign language recognition model and organized the data collection for Hong Kong sign language. The Nippon Foundation put up the money, while Google is responsible for the technology, developing the original concept and led the development of the AI-powered sign language recognition system.
Kwansei Gakuin University collected the data for Japanese sign language, or shuwa in Japanese, and provided insights into deaf culture in Japan.
The cross-culture development process allows users to enjoy the experience of signing in two sign languages, as well as the ability to to switch between the two in the game.
Throughout the game, users can collect various items after their signs are correctly recognized.
Users can learn signs from different common categories designed around themes ranging from placing orders in a cafe to finding accommodation while on a trip. SignTown also provides information about deaf culture in different regions.
Apart from hand movements and body motions, facial expressions and nonmanual features also play important roles in sign language, but this has mostly been overlooked by the past sign language recognition models, which focused mainly on hands and gestures.
Strict equipment requirements, including three-dimensional cameras and digital gloves for capturing sign movements, has made it even harder to popularize the technology in this field.
The team used machine learning models to track these significant features without special equipment or specific settings.
For example, PoseNet is used for human pose and gesture recognition, Facemesh is used for the mouth and facial expressions, and Hands Tracking is used for hand shape and finger detection. The sign language data was collected by native deaf signers from Hong Kong and Japan to ensure data accuracy.
“SignTown focuses on technical barriers in recognizing the phonetic features that are essential in sign language expressions,” Tang said. “Recognizing dictionaries and short sentence recognition will be our next step, and we hope these new technologies can bridge the gap between the deaf and the hearing.”
It released a beta version last week aiming to make improvements based on feedback from users.
The team aims to launch the first complete version of the game on the International Day of Sign Language on September 23.
(This article was published at The Standard on June 8, 2021: Education: Signs and wonders )