Continued from page 1
THE TWO MAIN CHARACTERS in Mission Rehearsal, the sergeant and the mother, are complex artificial-intelligence agents (software robots) capable of varying degrees of reason and emotion. They have a limited ability to hear what is said to them, analyze it and respond with speech, facial expressions, gestures or other actions. Other speaking characters, such as the squad leader, are bit players, completely scripted and incapable of interactive response.
Modeling human behavior of U.S. soldiers is hard enough, but building such models for foreign civilians or enemy troops presents new layers of difficulty for the researchers. For now, the sergeant is programmed to be level-headed and logical. He acts like a coach: If the officer orders some soldiers to go on ahead to the weapons depot, the sergeant will warn him that splitting his force is not a good idea. The Serbian woman is programmed to react emotionally.
“Her goal is to help her child,” explains Jon Gratch, an ISI project leader and assistant research professor of computer science, who worked on the emotional modeling for the mother. “When troops begin leaving, this triggers her anger.”
Visually, the characters are computer-generated avatars derived from motion-capture images of real actors. Wearing markers on their bodies, the actors bent their limbs and joints while a set of cameras captured their moves. The data generated in this way yields more natural body movements than would be possible using computer animation alone. Some hurdles remain, however. “If you haven’t captured the right motions, the character won’t have that [movement] in his repertoire. We’re working on a way to have the computer generate motion based on how the human body works,” Swartout says.
Jeff Rickel, research assistant professor of computer science at the USC Information Sciences Institute, notes that the characters not only have personalities and emotions, but also use speech recognition, are gaining an understanding of natural language and can synthesize speech. They are embedded in an entire virtual world.
The virtual sets and characters are not a one-time-only creation. These digital buildings and streets are the computer equivalent of Hollywood back lots, and so are the characters. They can all be tweaked, dressed up and recycled.
“We want competent, reusable virtual humans,” says Rickel, “and I believe we will have them within five years.”
ONE OF MISSION REHEARSAL'S greatest (and least expected) challenges was bridging the gap between the various team members’ cultures.
“It’s a rather crazy collaboration,” admits Lindheim, “and everyone had some skepticism at first. But once people got over the language barriers – the Hollywood-speak, the military acronyms and the computer jargon – they found they weren’t as different as they had thought. We have all developed a mutual respect.”
At first, it was difficult for both the military and the engineers to understand the importance of a good story. For a training simulation to fully engage trainees, its storyline must build; it has to involve emotions, a conflict and drama. “It can’t just be an event list,” says Swartout.
On the other hand writers, accustomed to linear stories, were challenged to create scenarios with many possible outcomes. And they had to write around serious technical limitations. For example, characters can’t touch each other. During one demo of Mission Rehearsal, a woman commented that it must have been created entirely by men, since the mother never caresses her child. The creative team had included women, and early drafts of the script had called for the mother to stroke her son’s hair. But it wasn’t to be. “The collision-detection mechanism isn’t good enough,” explains Swartout. “She might put her virtual hand right through her virtual son. For the same reason, we had to show soldiers getting into the humvee on the other side where you can’t see them touch it.”
The storytellers also wrestled with the way the Army works, according to ICT deputy director of technology Randy Hill, a West Point graduate. “All military operations have certain similarities,” he says. “There’s command and control. Soldiers follow the chain of command. I’d often find myself in a surrogate-Army role in meetings with both creative and technical people. Then, when I was with the Army, I had to communicate the technology to them.”
THE TWO MAIN CHARACTERS in Mission Rehearsal, the sergeant and the mother, are complex artificial-intelligence agents (software robots) capable of varying degrees of reason and emotion. They have a limited ability to hear what is said to them, analyze it and respond with speech, facial expressions, gestures or other actions. Other speaking characters, such as the squad leader, are bit players, completely scripted and incapable of interactive response.
Modeling human behavior of U.S. soldiers is hard enough, but building such models for foreign civilians or enemy troops presents new layers of difficulty for the researchers. For now, the sergeant is programmed to be level-headed and logical. He acts like a coach: If the officer orders some soldiers to go on ahead to the weapons depot, the sergeant will warn him that splitting his force is not a good idea. The Serbian woman is programmed to react emotionally. ICT executive director Richard Lindheim |
“Her goal is to help her child,” explains Jon Gratch, an ISI project leader and assistant research professor of computer science, who worked on the emotional modeling for the mother. “When troops begin leaving, this triggers her anger.”Visually, the characters are computer-generated avatars derived from motion-capture images of real actors. Wearing markers on their bodies, the actors bent their limbs and joints while a set of cameras captured their moves. The data generated in this way yields more natural body movements than would be possible using computer animation alone. Some hurdles remain, however. “If you haven’t captured the right motions, the character won’t have that [movement] in his repertoire. We’re working on a way to have the computer generate motion based on how the human body works,” Swartout says.Jeff Rickel, research assistant professor of computer science at the USC Information Sciences Institute, notes that the characters not only have personalities and emotions, but also use speech recognition, are gaining an understanding of natural language and can synthesize speech. They are embedded in an entire virtual world. | There is no other place in the world where Hollywood directors are working with computer scientists and colonels, not to mention [university-based] communications experts and cinema-television scholars.” |
The virtual sets and characters are not a one-time-only creation. These digital buildings and streets are the computer equivalent of Hollywood back lots, and so are the characters. They can all be tweaked, dressed up and recycled.“We want competent, reusable virtual humans,” says Rickel, “and I believe we will have them within five years.”ONE OF MISSION REHEARSAL'S greatest (and least expected) challenges was bridging the gap between the various team members’ cultures.
“It’s a rather crazy collaboration,” admits Lindheim, “and everyone had some skepticism at first. But once people got over the language barriers – the Hollywood-speak, the military acronyms and the computer jargon – they found they weren’t as different as they had thought. We have all developed a mutual respect.”At first, it was difficult for both the military and the engineers to understand the importance of a good story. For a training simulation to fully engage trainees, its storyline must build; it has to involve emotions, a conflict and drama. “It can’t just be an event list,” says Swartout.On the other hand writers, accustomed to linear stories, were challenged to create scenarios with many possible outcomes. And they had to write around serious technical limitations. For example, characters can’t touch each other. During one demo of Mission Rehearsal, a woman commented that it must have been created entirely by men, since the mother never caresses her child. The creative team had included women, and early drafts of the script had called for the mother to stroke her son’s hair. But it wasn’t to be. “The collision-detection mechanism isn’t good enough,” explains Swartout. “She might put her virtual hand right through her virtual son. For the same reason, we had to show soldiers getting into the humvee on the other side where you can’t see them touch it.”The storytellers also wrestled with the way the Army works, according to ICT deputy director of technology Randy Hill, a West Point graduate. “All military operations have certain similarities,” he says. “There’s command and control. Soldiers follow the chain of command. I’d often find myself in a surrogate-Army role in meetings with both creative and technical people. Then, when I was with the Army, I had to communicate the technology to them.”Photographed by David Strick
Related Story
Stars and Stripes
Other Features
Rock of Aging
A Voyage Through Time
Features -- Rock of Aging - Next-Generation War Games - A Voyage Through Time
Departments -- Mailbag - What's New - In Support - Alumni News - The Last Word