Communication is not entirely spoken. The learning system teaches the meaning of gestures, which the student must also be able to understand.

The Rapid Tactical Language Training System, created by the USC Viterbi School of Engineering's Center for Research in Technology for Education (CARTE) and partners, tests soldier students with videogame missions in animated virtual environments where, to pass, the students must successfully phrase questions and understand answers in Arabic.

Members of the U.S. Army’s Special Forces will be using the 80-hour system, now being completed at CARTE’s headquarters in the USC Viterbi School’s Information Sciences Institute.

To watch a video version of this story, go to: http://www.isi.edu/~jmoore/Mankin/MankinTLWeb.mov.

“Most adults find it extremely difficult to acquire even a rudimentary knowledge of a language, particularly in a short time,” said CARTE director W. Lewis Johnson.

“We’re trying to build an improved model of instruction, one that can be closely tailored to both the needs and the abilities of each individual student," Johnson said.

Cadets at the U.S. Military Academy at West Point studying Arabic tried an early version of the system in October 2003 and offered suggestions. December trials by enlisted personnel at Ft. Bragg were encouraging and led to making the material more accessible.

Johnson leads a six-person CARTE team that is spearheading the effort. The Defense Advanced Projects Agency (DARPA) and the Office of Naval Research are funding the work. The Rapid Tactical Language Training System is one of several instructional programs in a DARPA initiative aimed at developing active learning tools.

Part of the system, the “Mission Skill Builder,” resembles an intensive version of the language laboratory programs that have been in use for generations. In these, students imitate and practice words and phrases pronounced by native speakers.

“While our system is similar to drill-and-practice language programs that have been in use for some time, the Skill Builder incorporates some important innovations,” Johnson said.

These include:

• speech recognition technology that is able to evaluate learner speech and detect common errors;

• pedagogical agent technology that provides the learner with tailored feedback on his performance; and

• a learner model that dynamically keeps track of what aspects of the language the learner has mastered and in what areas the learner is deficient.

Along with linguistic skills, the program also instructs students in non-linguistic cultural matters of importance in communication.

“People don’t just communicate with words,” Johnson said.

“In face-to-face conversation, nonverbal behavior such as gesture, posture, gaze, head movements and facial expression play an important role in coordinating a successful exchange,” said ISI research scientist Hannes Hogni Vilhjalmsson, a specialist in modeling human nonverbal communication.

“Wrong interpretation of nonverbal cues or the wrong nonverbal responses can lead to serious misunderstanding and escalate hostility,” he said. “It is important to include all of these behaviors when teaching conversation skills in a foreign language.”

Exposing learners to realistic face-to-face situations, and training them to be culturally sensitive, prepares students to “become effective social players as well as speakers in the new language," Vilhjalmsson said.

Points covered in culture training include:

• social skills necessary to build rapport with local people;

• appropriate degrees of politeness to use in different social situations;

• how to disagree with someone without offending; and

• how to respond to offers to hospitality.

Gesture training includes common Arabic gestures that a Westerner might misinterpret (for example, Arabs may roll their eyes to mean “no”) and American gestures (such as thumbs-up) that an Arab might misinterpret.

The examination or application part of the training system, the “Mission Practice Environment,” is even more challenging, offering students an unscripted, unpredictable test of their mastery of these elements.

In this segment, students wearing earphones and microphones control a uniformed figure moving through a Lebanese village, complete with outdoor coffee bar. They meet animated Arabic speakers who (thanks to artificial intelligence-driven voice-recognition programs) can carry on free-form conversations.

“These AI figures can understand what the students say, if it’s said correctly - or won’t, if it isn’t. And they will respond appropriately,” said Johnson, CARTE’s director.

“In typical videogame fashion, the idea is to get to the next level,” Johnson said. “In this game, in order to get to the next level, the learner has to master the linguistic skills.”

The program already has features to adapt it to each individual user, noting consistent errors or difficulties, which can be targeted for extensive or remedial practice.

Researchers have completed approximately seven hours of the program. The full program will have about 80 hours of instruction and introduce perhaps 500 carefully chosen words of the “Levantine” Arabic spoken in Lebanon. If all goes as planned, the system may be deployed next year.

“We here in the Department of Foreign Languages are very excited about the Tactical Language Training System and the new capabilities that it can provide to military language learners, including our cadets,” said Col. Stephen LaRocca of the Department of Foreign Languages at West Point.

“This system allows learners to rehearse real-world tasks in the most realistic environment technology can provide,” LaRocca said. Working with CARTE on the project are the USC Integrated Media Systems Center; UCLA’s Center for Research on Evaluation, Standards and Student Testing; and Micro Analysis & Design Inc., of Boulder, Colo.

CARTE is headquartered at the Information Sciences Institute (ISI), which is part of the USC Viterbi School of Engineering. Besides Johnson and Vilhjalmsson, the ISI researchers involved include Stacey Marsella, Catherine M. LaBore, Dimitra Papachristou, Carole Beal, Nicolaus Mote, Shumin Wu, Hartmut Neven, Ulf Hermjakob, Mei Si, Nadim Daher and Gladys Saroyan.

For more information, go to the Tactical Language Project Home Page at: http://www.isi.edu/isd/carte/proj_tactlang/index.html.