Posted on Mon, 06/24/2013 - 15:23
When the Serious Game conversation turns to dollar and cents the chatter around the design becomes muted. Designing Digitally, Inc. knows the two are mutually exclusive and need to be addressed prior to investing some significant coin. We already know that stakeholders and decision makers must factor in the value of the investment with any learning programs created, and Serious Games are no different. We also know that we need metrics and artifacts to help inform our client of the true value of their educational product.
However, the metrics we can glean from a Serious Game are far more data rich than what we could expect from just a regular eLearning module residing in an LMS. This provides an opportunity to delve more fully into comprehending how the current design of a Serious Game is improving transfer of learning and where it may be hindering it.
The goal of a Serious Game is to impart knowledge, facts, concepts, and processes in a contextually meaningful way. The design provides opportunities for the learner to move through levels and/or reach goals. This typically occurs through interactions with the game elements such as other players, the environment, and/or objects. The standard goal and hope is for learners to apply their newly acquired skills and/or information outside of the game and in appropriate situations.
For example, a sales representative practices objection-handling techniques inside the game with various customers. The goal behind the Serious Game is to get the sales representative to actually utilize those techniques in their real-life sales calls. Are increased sales of the representative the only method of showing the game was worth the investment? It could be, but it wouldn’t be looking at all possible analytical options.
So where can you go to gather data? You can start with their Learning Management System (LMS). Any LMS can show you a few things about Serious Gamers, such as the gamers’ start and end times for the game as a whole or levels within the game and what was achieved (passed or failed). This can provide you with a surface comprehension of game use, but it still doesn’t tell you if game use was meaningful towards achieving the goals of the Serious Game. Gathering additional metrics on social interactions, routes that users took to achieve a goal or how a user leveraged an acquired game item to reach a goal or level would inform designers and stakeholders about the value of the investment. These new data points move away from just showing achievement, by actually capturing how something was achieved.
This additional information provides insight into the ROI discussion. Was the investment into this game worth it? Did the learners not only achieve what they should in the game, but did it also translate those achievements into actual accomplishments outside of the game? Given the previous example on objection-handling, what if the sales representative never used many of the techniques in real-life that they learned in the game?
If you only had start and stop times and pass/fail information you might be at a loss for why the Serious Game did not yield the result you had wanted. On the other hand, having the extra data points may tell you that cues from the game to socially interact with a peer or manager did not occur. Another example may be that perhaps the use of a game item, such as a clinical study to support a sales approach to objection-handling, was not leveraged. Determinations can then be made on whether the design of the Serious Game needs adjustment.
How can you get at this type of useful data? An upcoming option for gathering the more finite details from a Serious Gaming system is to use TinCan API which can be used in conjunction with an LMS. It does everything we just discussed with respect to data collection. It provides the ability to record any learning experiences of a learner or group of learners and their individual and collective learning paths. This translates into the ability to compare job performance to training data to determine effectiveness.
When deciding whether or not to go with a Serious Game make sure the conversation includes a balanced discussion around ROI, first from the perspective of the learner and then the investor. Ask what defines success for the learner when using the Serious Game. Then outline a strategy that will engage the learner and provide the/an opportunity to achieve each identified “success” in the game. Within that strategy, flag all aspects of the learner’s engagements and interactions that will help to build a data set for analysis.
For the investor side of the ROI, delineate what success looks like in a contextual sense. By this we mean, given the Serious Game, what do you expect the learner to be able to do with what they achieved? Is it to be more efficient, increase sales, heighten awareness, or decrease accidents? Revisit what was flagged for data collection and determine if more elements within the game need marked to ensure that all ROI specifications have attributable metrics.
As a final point, our discussion here focused mostly on what could be quantified from the Serious Game and from future performance of a learner. However, much is to be said about gathering qualitative information on the experiences of the users in the Serious Game. Aspects of motivation, self-efficacy, decision making, and other components of human behavior are difficult, if not impossible, to quantify as they are subjective and tend to be individually driven. These facets may be just as vital to the design of the Serious Game and the future performance of the learner.