USC engineers are perfecting a testing tool that detects the weaknesses and flaws in games – and may soon gauge the emotional involvement of players.

User testing is a critical element in creating a new game, but it remains a highly subjective and unstructured exercise.

“Traditionally,” said Tim Marsh, a postdoctoral researcher at the USC Viterbi School of Engineering’s Integrated Media Systems Center, “game companies hire teenagers and turn them loose trying to find flaws and gaps in the game,” which they report either verbally or in writing, along with their impressions.

This is neither systematic nor scientific, said Marsh, who is co-author of a conference presentation titled “Continuous and Unobtrusive Capture of User-Player Behavior and Experience to Assess and Inform Game Design and Development,” to be given June 26 at the Fun ’n Games 2006 Conference in England.

Marsh’s method analyzes Immersidata, a term referring to the record of commands sent to the computer by keyboards, joysticks and other controls collected in sync with a videotape recording of the player in action.

An IMSC-developed tool called ISIS (Immersidata AnalySIS) can “identify data of interest and index events within the videotape. For the game development application, ISIS can return indexed examples of six different kinds of occurrences, or “points” in the Immersidata/video record:

• activity completion points, when the player has finished a final task associated with a mission;
• task completion points allowing a researcher to go back over the performance of a task;
• break points, times when nothing seems to be happening, and the player is not moving;
• wandering points, somewhat similar times when the player is moving, but doesn’t select any objects;
• critical events. Some elements of the game are the hardest, and these can be pre-selected so that action leading up to an accomplishment or non-accomplishment can be studied; and
• navigation errors. Collisions with a wall or object potentially point to inadequate or poor design causing user disorientation.

During the tests, Marsh and USC computer scientist Cyrus Shahabi used a “serious” (i.e., teaching) game designed to instruct undergraduate students in human anatomy and physiology.

Though Marsh and the group tested the technique on a serious game, “the techniques are for testing all game genres – entertainment and non-entertainment,” he said.

Marsh is working on ways to capture other aspects of the game experience, including the emotional/empathetic elements. He recently wrote a chapter on the topic in a new book, “Gaming as Culture” (McFarland Press, 2006).

In addition to Marsh and Shahabi, USC computer science doctoral candidate Kiyoung Yang played a key role on the project, Marsh said. Shamus Smith of the University of Durham also participated and will present the paper at the conference.

The research was funded by the National Science Foundation and by a grant from the Presidential Early Career Award in Science and Engineering.