||Andrew Schwartz (Ric Evans photo)
The gleaming image is the only source of light in the tiny closet where Andrew Schwartz is crammed next to a monstrous black machine. His face reflects the glow of the monitor that flickers with the bizarre graphic.
It looks like Jupiter exploding.
Or a disco ball spinning with lasers.
Or Easter egg dye spraying from a blue balloon. In his mind, the colorful lines and blue sphere show the beginnings of how brain cells tell muscles to move.
The terminal drones as Schwartz turns dials and taps codes into the hunky computer to adjust the blues and reds and yellows. Tonight, like most nights this month, this neuroscience postdoctoral fellow has enclosed himself in the frigid closet that’s humming with air conditioners to keep the computer hardware cool.
Yet, he isn’t cold because he feels energized by his idea to illustrate a scientific concept in a way no one has before.
It’s nearly midnight, and Schwartz is still in high gear. It’s the only time he has access to this rare machine. He diligently types codes into the computer to translate research findings into a single, colorful visual image of a complex research theory—how neurons choose which motor signals to send to muscles. Based on the data entered, the tails of the purple lines darken. They look like fireworks instead of plain beams of light.
The resulting three-dimensional image—and the research behind it—wows scientists worldwide when it appears on the cover of Science magazine in September 1986 as the first computer-generated illustration of bioresearch findings.
Now, nearly two decades later, Schwartz’s research is poised to do more than wow scientists. It has the potential to dramatically improve the lives of patients with paralysis, amputated limbs, and even Lou Gehrig’s disease. His work at the University of Pittsburgh is making once-impossible feats seem, literally, within reach.
In Schwartz’s spacious research lab, a Rhesus monkey with a prosthetic arm reaches for a grape.
The shoulder joint shifts so the arm swings in a diagonal motion, and the elbow extends outward as the monkey, named Pierce, grabs the fruit. Then, he bends the robotic arm to bring the treat to his mouth.
The movement is smooth and almost looks natural. Pierce is moving the anthropomorphic arm just by thinking about it. He controls the out-of-body device in the same way that you blink, or walk, or roll your tongue.
A bioengineering graduate student holds out another grape for the monkey to eat. Neurons in Pierce’s brain zip and zap as he sees the fruit and thinks about grabbing it. Thousands of brain cells in the motor cortex are firing at different intensities to indicate how he should move his arm.
The monkey’s real arm is bound to his side, but an implantable device, developed by Schwartz, picks up electric zaps from 50 or 60 neurons that normally control the primate’s real arm. The signals zoom through a matrix of wires to a terminal where a postdoctoral fellow is monitoring the monkey’s brainwaves on an oscilloscope. On the screen, a green line spikes and dives and rises back to the surface as the cells fire.
The signals are also being channeled through a desktop computer that has been programmed with an algorithm based on the research Schwartz illustrated on the cover of Science two decades ago. The mathematical formula tallies the incoming signals to figure out the preferred movements of the group of cells and ultimately directs a separate, plastic robot arm to move in the appropriate direction.
It all happens in a few seconds. Pierce is already sucking on the grape.
The arm experiment, led by Schwartz, now a professor of neurobiology in Pitt’s School of Medicine, is the first to successfully translate neuronal signals into direct robotic control that mimics human movements.
For more than a decade, technology to control a computer cursor with brainpower has been available. Some paraplegics use brain implant technologies to check e-mail and surf the Web. Although it has allowed them to perform an independent task, the output is only produced in the virtual space of a two-dimensional computer screen.
Now, Schwartz is exploding science into a third dimension. With the robotic arm, he’s on the verge of developing physical limbs that patients can control using their own brain power.
“His lab is probably the best in the business—in the country—right now,” says Dan Moran, assistant professor of biomedical engineering at Washington University in St. Louis. “He’s got the best control. He’s got the best results.”
Other scientists who are doing similar research can only move their prosthetic instruments in one or two directions, called degrees of freedom, says Moran, who worked with Schwartz from 1995 to 2000 at the Neurosciences Institute in California.
Schwartz’s technology can manipulate the robotic shoulder and elbow with a total of four degrees of freedom, just like a human arm. The movements are also faster and more accurate than what other researchers have produced.
Moran says Schwartz’s success isn’t surprising, considering the large number of scientists in this burgeoning field of neuroprosthetics who are basing their research on the neuroscience papers that stemmed from Schwartz’s work with Apostolos Georgopoulos, his postdoctoral advisor at Johns Hopkins University in Baltimore.
The two men neurologically recorded natural three-dimensional arm movements in monkeys. This provided direct evidence for Georgopoulos’ theory about how the intensity of neuronal signals have preferred directions and how these responses can be used to predict movement. Previously, only restrained movements in two dimensions had been analyzed. It was this research that led to the cover image on Science magazine in 1986.
“It was like a revelation,” Georgopoulos says in recalling the three-dimensional visual image that conveyed their research theory. “Andy’s the most original and most motivated and dedicated person that I have seen.”
After Schwartz earned his postdoctoral fellow distinction at Johns Hopkins in 1987, he pioneered an important theory about time trajectories as a staff scientist at the Barrow Neurological Institute of St. Joseph’s Hospital in Arizona. There, he figured out how to accurately predict arm movements by analyzing trajectories—or signaling paths—in painstaking millisecond-by-millisecond time frames. At the time, scientists were only looking at the averages of the neural activity. Schwartz discovered valuable information about the increases and decreases of activity within these small time frames, which enabled him to accurately predict physical movements.
“Different techniques other colleagues are using all stem from two things,” says Georgopoulos, now director of the University of Minnesota Center for Cognitive Sciences. “One is the population vector idea that we published in 1983, and the other is Andy’s real contribution on retrieving the time trajectory of the movement that we published in 1988. This whole field of trying to control prosthetic devices and moving robots, it goes back to those key applications and ideas.”
In the early 1990s, Schwartz—a half-neuroscientist, half-engineer who, as a child, took clocks apart and figured out how to put them back together—started building the electrodes, computer hardware, and robotic arm that are part of his experiments today. Progress was slow at first but rapidly increased as computer technologies improved.
He continued to engineer the neural prosthetics in 1995, when he became a senior fellow at the Neurosciences Institute and research professor in the Department of Chemical, Bio and Materials Engineering at Arizona State University.
In 2002, Schwartz was appointed to his neurobiology professorship at Pitt, in addition to being named a professor of physical medicine and rehabilitation in the medical school and professor of bioengineering in the School of Engineering. He also joined the faculty at the Center for the Neural Basis of Cognition, a joint research effort between Pitt and Carnegie Mellon University. A year later, Schwartz began testing the technology with monkeys.
Corris Hammock glides around Schwartz’s laboratory, a vast white space with high ceilings and enormous windows that offer a glimpse of the Hot Metal Bridge spanning the Monongahela River.
He steers his electric wheelchair with his chin, using a special joystick. Hammock is a quadriplegic who lost the function of his limbs more than 20 years ago when he injured his spinal cord in a somersault accident.
To Hammock, the sterile and uncluttered room with special equipment terminals looks like a futuristic lab from a Star Wars movie. He’s visiting to observe some of Schwartz’s technologies, which could potentially benefit him or his disabled clients. Hammock is a counselor at the University of Pittsburgh Center for Assistive Technology in Oakland, which provides tools for patients with a broad range of physical disabilities.
Schwartz leads Hammock to the area where one of the monkeys is eating with the assistance of the robotic arm. As the robotic arm bends to bring a treat to the primate’s mouth, Hammock thinks that something like this would enable him to feed himself, which he hasn’t been able to do in two decades.
“I know it could really revolutionize the ways that people with disabilities do things,” he says. He not only envisions feeding himself, but fishing or target shooting, two of many activities he enjoyed before his accident. He imagines pulling a shirt from a shopping rack to get a better look at the color and style. He imagines ringing a doorbell or pushing an elevator button. He imagines opening a Christmas card on his own.
It will take time, but if Schwartz’s robotic arm proves viable, the technology will undoubtedly help people with paralysis, amputated limbs, and possibly even amyotrophic lateral sclerosis, or Lou Gehrig’s disease, a neuromuscular illness that weakens and eventually destroys motor neurons.
Schwartz plans to begin testing the robotic arm with human patients in three to five years. The next challenge will be to see if he can control an anthropomorphic wrist and fingers. Currently, the robotic arm has a three-pronged gripper at the end. Human hands have 22 degrees of freedom—more than five times the degrees of freedom that are possible in a shoulder and elbow.
“It seems like it has more potential than I can fathom,” Hammock says after visiting the lab. “I entered another world that revolves around people doing something that’s never been done before. You could feel it in the air.”
For Schwartz, the thrill of his success is that he’s figured out a little bit more about the brain.
“I’ve always been interested in mechanics, how things work,” he says while sitting in his office, which has floor-to-ceiling bookshelves stacked with hundreds of science journals and texts. “How does a complex system solve a particular problem? How would you design a system to do it?”
These are the questions this 49-year-old asks himself during the idle moments of life—like when he’s pulling legs on his daily 5-mile run or when he’s turning the steering wheel during his drive to the lab.
The engineering side of his mind analyzes human movements as “really elegant solutions,” while his creative side thinks about how he could use the same approach. “When we move and reach, it’s smooth and graceful,” he says while he stretches his arm across his desk. “You understand my emotions by the way I move.”
This self-described “naïve Minnesota farm boy” used both the analytic and creative aspects of his intelligence to construct the brain-powered robotic arm—by analyzing the brain’s solution of moving the arm and then mimicking the motion.
Schwartz calls his innovative side his “pragmatic creativity,” a concept of thinking practically, but beyond the norm. He encourages this type of thinking in his students and in the Journal of Neural Engineering, a publication for which he serves as coeditor in chief.
It’s the type of thinking he used 20 years ago when he proudly designed the computer graphic for the cover of Science.
It’s the type of thinking that spills ideas off his tongue as he talks about how the brain communicates with itself, how the timing of neural signals affects that communication, and how new technologies will soon allow him to bypass monkey experiments and directly test human cognition.
It’s the type of thinking that keeps Corris Hammock—and others like him—dreaming about what could be.