After suffering a devastating brainstem stroke at age 30 that left her nearly completely paralyzed, Ann had lost the ability to speak for 18 years. Now, thanks to AI and an innovative new brain implant developed by UC San Francisco and UC Berkeley researchers, this Canadian woman is speaking again.
Not only does the implant use Ann's brain signals to generate speech, but it also animates a digital avatar with facial expressions, mimicking the natural nuances of human communication.
Led by Dr. Edward Chang, chair of neurological surgery at UCSF, the team implanted an array of electrodes on the surface of Ann's brain. The paper-thin device intercepts Ann's brain signals as she attempts speech, translating them into words, phonemes, and even facial expressions.
This breakthrough study, published on August 23, 2023 in Naturex, represents the first time speech and facial movements have been synthesized directly from brain signals. It's a huge leap forward for brain-computer interface (BCI) technology and the goal of restoring more natural communication for people with paralysis.
Chang has spent more than a decade advancing Brain-Computer Interface (BCI) technology. In 2021, he and his research team developed a "speech neuroprosthesis" that allowed a man with severe paralysis to communicate in full sentences. The technology captured signals from the brain directed toward the vocal tract and translated them into words that were displayed as text on a screen. This breakthrough marked the first time that it had been demonstrated that speech-brain signals could be decoded into complete words for an individual living with paralysis.
When Ann suffered her brainstem stroke, she lost the ability to move her muscles and communicate. Through intense therapy, she slowly regained limited facial expressions and some head and neck motion. She could communicate painstakingly slowly using a head-controlled typing device. But technologies lacked the sophistication to decode Ann's brain signals into fluent speech.
Hearing about Dr. Chang's work, Ann enthusiastically volunteered for the study last year. Chang's team recorded her brain patterns as she attempted to recite words, training AI algorithms to recognize the signals for various speech sounds.
Rather than whole words, the AI decodes phonemes - speech's basic building blocks - making it three times faster and more versatile. Already the system transcribes Ann's attempted speech into text at nearly 80 words per minute, far beyond her old communication device's 14 words per minute.
Using Ann's wedding footage from 2005, researchers used AI to reconstruct her unique vocal tone and accent. They then used software developed by Speech Graphics to create a personalized digital avatar that mimics Ann’s intended facial expressions in real-time. Now when Ann tries speaking, the avatar seamlessly animates and voices her intended words.
The work of Chang and his team represents more than just a technological breakthrough. It signifies hope for countless people suffering from severe disabilities. Their next goal is to develop a wireless version of the BCI which eliminate the need for the user to be physically tethered to the system.
For Ann's daughter, this breakthrough means finally hearing the voice of the mother she's never known outside of a computerized aid. Bringing profound independence and self-expression within reach, the technology represents hope for Ann and countless others robbed of speech by paralysis.