On the screen, computer-generated characters shrug, wink, nod, wave or cross their arms as they follow one’s every move with an attentive gaze.

Meanwhile, a USC-developed system module called “Social Puppet” is pulling the strings.

Once a given character is designed, a simple set of standard commands orchestrates a whole range of nonverbal expressions. The same commands work for any other character in the game.

“Human communication is only partly verbal,” said Hannes Högni Vilhjálmsson of the USC Information Sciences Institute, who designed the game.

He calls the software an “engine” to generate visual social behavior, and will present it at the AAAS annual meeting in St. Louis Feb. 16-20.

Vilhjálmsson is among the builders of a set of ISI-created videogames called “Tactical Language and Culture” that the armed forces now use to teach language and customs to soldiers. Hundreds of soldiers have trained with “Tactical Iraqi,” while a “Tactical Pashto” is being readied for Afghanistan.

In the games, a person controls a figure representing themselves, who interacts with other characters animated by artificial intelligence, a specialty at ISI, which is part of the USC Viterbi School of Engineering.

“To introduce players to a culture that is unfamiliar to them,” Vilhjálmsson notes in his presentation, “it is important to have them both observe nonverbal behavior that reflects the culture and have them be able to perform … appropriate behaviors in return.”

Additionally, “when having a conversation face to face, people rely on … spontaneous nonverbal cues such as gesture, gaze and head movement. This is even more critical when trying to have a conversation in an unfamiliar language.”

Vilhjálmsson said students need feedback from the game to know how well they are communicating. “Puzzled or offended expressions are much more intuitive clues than, say, a printed message saying ‘your words [are] not understood’. ”

ISI’s Lewis Johnson, director of the Tactical Language and Culture Project, said, “People naturally tend to rely heavily on nonverbal communication when they are learning a foreign language, to make up for their lack of verbal knowledge.

“But nonverbal gestures can sometimes be a source of confusion, since people in different cultures tend to employ different gestures. So we believe that it is important to include nonverbal communication in Tactical Language and Culture, both to promote verbal learning and to train people to communicate effectively face to face with people in other cultures.”

Vilhjálmsson’s work on Social Puppet was financed by DARPA and carried on at ISI’s Center for Advanced Research in Technology for Education (CARTE).