Researchers receive $1.5 million grant to develop communication technology

Communication is deeply personal. Whether it’s an accent, a go-to phrase or regional colloquialism, nearly everyone has a unique way to vocally express themselves.  

So, when Stephen MacNeil heard about research aiming to make Augmentative and Alternative Communication devices easier to personalize for people who struggle with speech, he knew he had to get involved.  

“The kinds of things you’d want to say in the bathroom versus the kitchen are going to be very different,” MacNeil said. “Being able to adapt to those communication options in real-time is one of the biggest things that we’re doing.” 

High-tech AACs are typically smart tablets with grid displays of words and phrases. Users select the phrases they want to communicate, and they’re played out loud. 

MacNeil is working with researchers Slobodan Vucetic and Eduard Dragut to create AAC devices that understand context by integrating them with generative artificial intelligence.  

Through a $5 million grant from the National Institute on Disability, Independent Living and Rehabilitation Research, the trio is working with Temple’s Human-Computer Interaction Lab and three other universities to develop adequate AAC technology.  

Temple’s $1.5 million share of the grant will go towards the engineering and technology components of a new Rehabilitation Engineering Research Center located at the University of Arkansas. 

As a HCI and AI researcher, MacNeil’s collaboration with Vucetic and Dragut at Temple in 2021 was natural. Vucetic and Dragut were already working on improving AAC devices with generative AI, through funding from the National Science Foundation. 

During the past five years, MacNeil, Vucetic and Dragut have heard users discouraged by difficulty customizing AAC devices to their unique communication and the necessary reprogramming every time they change location. 

“It was not very usable,” MacNeil said. “A lot of people would stop using the communication device, which limited their ability to communicate.” 

Context-aware AAC technology harnesses AI capability — audio, visual, textual processing and more — to understand where a person is. When the tech knows a user’s environment, it can suggest the things they’d probably like to communicate. 

The group co-designs AACs with user preferences, which vary on privacy and device capability. Components that require device listening, like audio processing and other context-aware tools in GPS and Bluetooth, are possible technologies that can increase AAC device capability.  

However, users may not be willing to trade privacy for increased capabilities. 

“We hear from users we work with that [they] don’t want AI to take over,” Dragut said. “They still want to have a voice, and AI be sort of in the background.” 

Undergraduate and graduate students under MacNeil at Temple’s HCI Lab use “Wizard of Oz” methodology, in which undergraduate students develop manually controlled prototypes and test on users who believe the technology is automated before PhD students develop the AI capabilities. 

The prototypes developed at Temple will be sent to researchers conducting clinical testing with real AAC users at the other universities sharing the grant. MacNeil’s team will modify their models according to feedback that they are given. 

“I was a psychology student, and I think that demonstrates just how interdisciplinary this whole concept is,” said Kate Hamilton, director of operations at the HCI Lab. “It’s not just computer science people. It takes a whole community.” 

The team has already seen success integrating AI with AAC tech in past testing. Joel, an AAC user with cerebral palsy who the team previously worked with, is understood most of the time but struggles with certain words. He used the devices to speed up his communication. 

Joel faced the same time-consuming problems as other AAC users, and the new improvements were revolutionary. 

“This device was sort of listening and reproducing what he was saying in a smoother way,” MacNeil said. “When he heard the first one, he shrieked. He said, ‘I’ve been waiting for this my entire life.’”