Scientists from the University of California at San Francisco (UCSF) and the University of California at Berkeley have developed a groundbreaking system that enables a woman with severe paralysis from a brain stem stroke to communicate using a digital avatar. The system, which can synthesize speech and facial expressions from brain signals, has an impressive speed of almost 80 words per minute. The results of the study were published on August 23, 2023 in the journal Nature.
The system is based on brain-computer interface (BCI) technology, which has been the focus of Dr. Edward Chang, the chairman of UCSF’s Department of Neurosurgery, for over a decade. Dr. Chang hopes that this technology will soon receive FDA approval and become accessible to patients who have lost their ability to speak due to strokes or other causes.
To enable the system to function, Dr. Chang implanted a slender rectangular device containing 253 electrodes onto the woman’s brain, specifically over the regions responsible for speech. These electrodes register brain signals that control the muscles of the tongue, jaws, larynx, and face. The signals are then transmitted via cables to computers, where they are processed using artificial intelligence.
Over several weeks, the participant collaborated with the research team to train the artificial intelligence algorithms in recognizing her unique brain signals for speech. She trained the system by repeating various phrases from a conversational dictionary comprising 1,024 words.
Rather than teaching the AI to recognize complete words, the researchers developed a system that decodes words from phonemes. By adopting this approach, the computer only needed to learn 39 phonemes to decipher any English word. This enhanced the system’s accuracy and tripled its speed.
To produce her voice, the woman utilized speech synthesis software, which was adjusted to match the sound of her voice after the stroke.
The woman’s avatar appears lively due to specialized software that replicates the movements of facial muscles. Powered by artificial intelligence, the avatar is synchronized with the woman’s brain signals, enabling it to simulate lip movements, tongue positions, and facial expressions. The scientists hold high hopes that this technological breakthrough will restore people’s ability to communicate and interact with the world, even after experiencing severe strokes.
However