Once Is Not Enough: How to Playtest Custom Learning Games

Once is Not Enough

I believe playtesting is a crucial, and often overlooked, part of learning game design. It takes multiple iterations to determine the right combination of game mechanics and game elements for your target learners. Whether you are an experienced game designer or an instructional designer trying to design a game for the first time, your game will benefit from multiple rounds of playtesting.

Of course, if we are going to tell everyone else to playtest their games, we have to do so as well! The following is the step-by-step story of how we created and play-tested a recent learning game for a client.

Example: Serious Game for Healthcare Workers

We recently created a game called “Five Star Facility,” targeted towards environmental technicians who work in healthcare settings. These technicians clean the patients’ rooms and other areas of a hospital or long-term care facility. It took us four iterations to get the game we wanted and players could most learn from.

We reached a great game in the end, but it took good playtesting and iteration to get us there. Our stellar design team of Amanda Gentry, Matt Kroeger, Kristen Hewett, and Erika Bartlett did a terrific job!

Version One – Let’s create something “sort of like Clue”

This version has similarity to a commercial game called Clue. The design team felt the target learners would be pretty familiar with Clue’s  rules and core dynamics (exploration, collection), and they wanted a game that learners could quickly learn to play. Gameplay was competitive. The game goal was to be the first person to collect all the room tokens, which represented all the categories of information players needed to learn and remember. Players rolled a die to determine how many spaces they moved on the board. Each space corresponded to a different category of environmental protection/cleaning. Players had to answer questions related to whatever category they landed on. Similar to Clue, they had to go into each room on the board. Answering a question in the room earned them a token for that room. The first player to earn all the tokens won the game.

First version of five star facility game.

First version of five star facility game.

It wasn’t a terrible game design… but it was just okay.

Problems With the First Design:

  1. The game could get sort of long if people were rolling lots of low numbers.
  2. The designers made the game competitive. In the real-world, environmental technicians should behave cooperatively with each other and with the healthcare team as a whole.
  3. This first rendition ignored the “why” of the environmental tech’s job and didn’t help them see the connection between what they do and how the healthcare facility gets dollars to stay in business. Survey ratings determine the reimbursements healthcare facilities receive from Medicare. If your facility’s aggregate survey ratings are only three stars, you do not receive the same dollars as a facility who received five star ratings.

Version Two – Scrap the Clue idea. Let’s race to the finish.

The icons are gone, the die is gone, and we have a path we’re traveling and monetary targets to reach. This version stunk. It was boring and tedious to play. Players simply took turns drawing cards to try to reach the target dollar amount. They worked together to answer the questions, but when the designers switched from competition to cooperation, they failed to include game mechanics that created any conflict or tension within the players. There was no “Lose” state or no really bad things that could happen. This version was quickly ditched.

Version Two: Lots and lots of dots.

Version Two: Lots and lots of dots.

Version Three – Bring back the game icons. Add in a progress tracker.

Version three was much better! The team latched on to the realization that five star ratings led to better reimbursements. Now players had to secure at least $70K in reimbursements to win… and mistakes would push their survey ratings downward. This was better, but there was still a serious flaw. Players’ dollars didn’t go down when they made mistakes; only their survey ratings did. In the real world, these are tied together. We also discovered as we played that we needed to better write our questions to eliminate ambiguity of responses. On the plus side, the discussion team members did before deciding on a correct response was phenomenal. Lots of learning happened in these discussions.

Version 3: We’re getting closer to the final product.

Version 3: We’re getting closer to the final product.

 Version Four – We have a winner!

The final version of the game was the winner. Look at how we tied together survey ratings and reimbursement dollars. Players start with a 1.5 star rating and $30K in reimbursement dollars. To move to the right and earn more dollars, they have to enter a room and respond correctly to that category’s question. They still roll a die to move a team token around the board. If they land on a space outside of a room, they have to answer a question that corresponds to the icon they land on. A correct response allows play to progress to the next player with no adverse event. An incorrect response forces players to move to a lower survey rating. If they hit the zero starts spot on the game board, the game is over and the team loses. If they earn $10K from every “room” on the game board and achieve at least $70K in reimbursement, they win.

Version four: By George, we’ve got it!

Version four: By George, we’ve got it!

Lessons for learning game designers:

  1. Make sure your choice of a competitive game or a cooperative game mirrors the real-world environment. Do not have people competing in a learning game if their real-world context requires cooperation or collaboration to be successful.
  2. Be aware that competitive games do not tend to be as influential of learning experiences as cooperative ones do. In competitive games, only one person or team wins. The “losers” can disengage from the experience entirely if it is not managed well.
  3. Make sure the game mechanics (rules) and game goal complement – or at least do not detract – from your real-world situation.
  4. Make sure your game includes enough “tension” in it to keep things interesting. Interesting translates into “fun.” If there are not realistically significant odds of losing the game, it becomes boring to play.
  5. Don’t be content with the first version of your game; it will not be the best version.
  6. Don’t playtest once; identify changes to make, and then fail to playtest to verify those changes improve the game play and learning experience. You have to test every time you make a change.

Game or Mini-Game: Which Should You Choose for Your Training Solution?

You are all in. You want to incorporate games into your learning experience. Do you go with one large game as THE learning solution or do you incorporate several smaller games… what we call “mini-games?”

A large-scale game can provide learners with a powerful, immersive learning experience. However, such games can be complex to design and develop, and they typically require lots of personnel hours to produce.  If you are new to the learning game design arena, they can be an intimidating task.

Sometimes, a small “mini-game” can provide you with the level of engagement you want while not requiring as much time and effort to produce.  Consider using a mini-game if these things are true:

  • You want to teach or reinforce a single learning objective (e.g. Compare 5 products, Distinguish between 3 things, Classify 4 personality types, Recommend the right settings, etc.). If you have more than a single objective in mind, a mini-game is not a realistic choice.
  • You want total playing time to range between 5 and 15 minutes of time and no more. It’s not a “mini-game” if it takes an hour for people to play.
  • You are not seeking a game that people will play over and over again. Most mini-games are structured as matching activities, puzzles, or quick scenarios. Once the problem is solved, people typically don’t want or need to replay the game, particularly if you design it so they cannot complete the game without getting things right.
  • The game functions with other components in your curriculum and is not the only way you are teaching something. Mini-games tend to be a one-and-done activity so they function best when combined with other things. A mini-game might be a great pre-work activity or a good activity to include in an online learning event or eLearning course.

Let’s compare a situation where a mini-game was the right learning solution with a situation where a more comprehensive serious game was needed.

1. Incident Investigation

These thumbnail images link to different mini-games. They are meant to be used in one of two ways: as part of a larger endeavor to teach incident investigation and/or as a quick reinforcement of basic concepts related to incident investigation. Each mini-game has a single learning objective. (A full-blown game might have several.)


In The Elevator Game the game goal is for learners to get 11 people to the right location in a building by 3 p.m. The learning goal is for them to accurately identify the activities associated with the five steps of the incident investigation process. Play the mini-game.


In Late for Lunch the game goal is for learners to get to lunch before starvation sets in. The learning goal is for them to accurately identify the information they need to gather regarding a recent accident that occurred at their job site. Play the mini-game.


In Making Gold, the game goal is to escape from the evil alchemist’s laboratory. The learning goal is to accurately distinguish between an incident, an accident, and a near-miss. Play the mini-game.

2. Product Launch

We recently completed a major curriculum design and development project associated with the launch of a product. The curriculum included pre-work components, and a two-day live event. One of the pre-work activities was a mini-challenge called Making Fuel. Learners had to correctly compare 7 competitors. The second game was a full-scale learning game. It used Knowledge Guru’s game engine to create a four-topic game that helped learners master product basics before attending the live launch workshop.


The screen grab above shows a mini-game that has the same functionality as the Making Gold game. The game goal is to get fuel into the car. The learning goal is to match competitor weaknesses to a product.


This screen grab shows a full-blown game that uses Knowledge Guru game engine. In this game, the players gain product knowledge through game play. The game goal is to win the Racing series by completing four different races. The learning goals included: 1) Match the features of the product with related competitive advantages, 2)  Match the competitive advantage with related benefits or savings, 3) Select the differentiating features and benefits of the product versus selected competitor products, and 4) Select differentiating features and benefits of the product versus selected competitor products.

6 Lessons from the Trenches of Digital Learning Game Design at #Trg14

Lessons from the Trenches of Digital Game Design

Bottom-Line Performance has done simulation and learning game design for table-top/live events for many years. However, our foray into digital game design has only been happening for the past three years. When we started, we found lots of books and articles on game design – but not much on learning game design. We leveraged wonderful books from game designers such as Tracy Fullerton and Brenda Brathwaite and gleaned from their experiences designing games, but we didn’t have a ton of peers writing tomes on learning game design. (Exception: Karl Kapp’s book, Gadgets, Games, and Gizmos for Learning came out in 2007. It’s a wonderful book, but it’s not really a how-to guide to creating learning games. His most recent book, The Gamification of Learning and Instruction Fieldbook has more how-to guides within it.)

There are some similarities between learning design and learning game design… but even more differences. Here is a summary of six lessons we’ve learned. We’ll present these – with more detail – at sessions we’re doing at ASTD TechKnowledge later this week as well as at Training 2014 in early February. You can get a sneak peak at the slides we’ll use (and the example) on Slideshare.

1. You need game content – even at your first prototype.

This might sound obvious, but if you have done agile design before, you may have designed HOW a learning interaction is going to work while including only placeholder content in it or Greek text. You cannot do this with a game. You have to have realistic content (e.g. an actual scenario and realistic choices for a player to make) or you cannot assess the fun factor and learning efficacy of the game idea. Trust us on this. We made the mistake of trying to design a game interaction with only place-holder content. People played the prototype and then told us, “Well it might be fun but I can’t really tell without seeing the actual game content.” Once we played, it was like the Mr. Obvious show. However, BLP has lots of smart people and we didn’t recognize this issue until we programmed an initial prototype that we called “Story Shuffle.” We got smart and re-did things. Here’s a later view of the same game, now called “Late for Lunch.” For those who are curious – we used a tool called Construct2 to create the game. You can embed games into course authoring tools such as Lectora or Articulate Storyline.

2. Aesthetics and theme dramatically affect desire to play. They literally can be game-changers in terms of people’s interest in what you create.

Again this seems obvious… but aesthetics are HUGELY powerful. They can take content that an ordinary person would NOT find exciting and make you want to play just because the game is so aesthetically cool looking. You might not be excited by the topic of incident investigation but you might be far more excited to go into an evil alchemist’s laboratory and earn your way out by making gold out of iron. Check out this game to see what I mean.

3. Fantasy has high appeal – even to “corporate” learners. It’s worth fighting for.

Bean counters can be skeptical of fantasy – it can seem frivolous or too fanciful for work. However…that is sort of the point in making someone intrigued enough to want to play a game that would otherwise be rather ho-hum.

Here’s ho-hum.


Here’s pretty fun:


4. Most players need help figuring out how to play – but typically won’t opt for it if given a choice.

This lesson is a critical one. Some learning games – in fact, many learning games – require some “show” on how to play to minimize the learner’s cognitive load. You don’t want them to spend so much mental energy figuring out the mechanics of the game that they fail to learn anything. However, when you design a tutorial level of play, if players get a choice, they will often OPT OUT of completing it…because they don’t want to take the time. We’ve learned not to let players have a choice and to require them to go through the tutorial. No, they won’t want to. Yes, it will end up maximizing their enjoyment of the play experience if they do. Either incorporate a “training level” or an actual tutorial into the game unless the game’s mechanics are very, very easy to understand and intuit.

5. Rules and game complexity need to be proportional to the amount of time people will spend playing the game.

If you are designing a multi-hour play experience, you can incorporate lots of game elements and mechanics (aka game rules). If you are trying to create a 10-minute to 60-minute play experience, you NEED TO KEEP IT SIMPLE. Lots of complexity can create a very fun GAME experience, but it has a negative impact on the LEARNING experience. As you play test your game during development, you need to ask both of these questions:

  • How engaged were you in play?
  • What did you learn by playing?

If the game has lots of clever elements and mechanics, you can get very positive responses to the first question – but poor responses to the second.

6. Scoring is the hardest element to get right – and requires far more time than a novice designer will probably assign to the project plan for it.

I created the game Formulation Type Matters four years ago. It was my first digital game (and a finalist this year in the Serious Games Challenge – hooray!). I allocated 8 hours to define the scoring for this game. We actually spent well over 40 hours figuring out the scoring – mostly because I didn’t have a clue what I was doing when I started. I’ve learned a lot since then, and I am now more careful to think through the scoring to make sure it’s relevant to the skill I’m trying to teach, meaningful to the player, motivating (rather than de-motivating to the player,) and, frankly, easy to understand. I also know that it is probably going to take more than 8 hours to figure out the scoring on a game unless the game is super-simple.

Visit Us at ASTD Techknowledge!

2014TK-Sponsors_SilverThere are several ways to see the BLP team at ASTD TechKnowledge:

1. In the Expo: We’ll be showcasing our Knowledge Guru game engine, and releasing the new Enterprise Edition, in Booth 303 of the Expo. Learn more.

2. In my session: I will give Lessons from the Trenches of Digital Game Design on Wednesday, January 22nd from 1:15 – 2:15 pm. Learn more.

3. In a hands-on demo: I will partner with Steven Boller to give Powerful Learning Games You Can Build in a Day on Thursday, January 23rd from 12:55 – 1:25 pm. Learn more.

4. In Leanne Batchelder’s session: Leanne will deliver Gamify Online Safety Compliance Training: A Roche Case Study, on Thursday, January 23rd from 3:15 – 4:15 pm. Learn more.


This Week on #TalkTech: Live TED talks with Spin, Serious Games for Healthcare, and Mobile Device Management

#TalkTech is the “flipped” approach to Twitter chats. We publish all the topics a few hours before the chat so you can show up at 3 pm EST / 12 pm PST on Thursdays ready to discuss.

We’re never at a loss for new tech news, apps, and interesting blogs to explore in our weekly chats. Take a look at what we have planned for this week:

Spin enables synchronous online learning with colleagues.

Topic 1: What are the benefits of learning from media like TED talks using “Spin”?

TED talks are a wonderful source of ideas, inspiration, and informal learning… yet the ideas become even more powerful when shared with others. With Spin, a new website by Net Power & Light, you can now watch a TED talk with friends and coworkers, commenting and sharing in a virtual environment. Technology so often focuses on solo learning, and Spin offers a unique way to reconnect the community element.

Learn More About Spin

Topic 2: How can serious games and simulations benefit learners in healthcare?

We talk a whole lot about the power of games here on #TalkTech… because they are a highly effective use of technology for learning. The video below is a 3D simulation used to train nurses on treating trauma patients in the ER. It was designed by Designing Digitally to meet the requirements of SCORM, yet we think it could be even more effective delivered on a tablet with the help of Tin Can API. What type of expertise does it require to create a simulation like this?

Watch a demo of the healthcare simulation/serious game

Topic 3: Is a mobile device management system necessary to implement mobile learning inside an orgaqnizatin?

What role does the company play in monitoring and managing all of those mobile devices it deploys? It can be pretty tricky to manage a BYOD program effectively, too. For this reason, some organization are turning to a Mobile Device Management System to help oversee their mobile initiatives. The article below offers a nice summary of the MDMs out today. What role should they play… and do you think they are necessary to implement mobile learning?

10 BYOD mobile device management suites you need to know about

If you’re new to Twitter chats, don’t forget about awesome tools such as that automatically save the hashtag and help you focus on the conversation!

How We Externally Tested our Learning Game Design

For those of you who have been following our occasional case study on learning game design, the end is near. The finish line of game development on our game A Paycheck Away is so close I can symbolically touch it. By month’s end, we will call development done and move into production of the game.

As a refresher – here’s the initial game board design followed by the current design. We’ve come a long way since the first prototype!

Learning Game Prototype
Our first game board prototype


A Paycheck Away learning game - game board
Our final game board design!

Our external play test went very, very well. We got the highest compliment you can bestow on a game. Testers rated it 4+ out of 5 for “flow,” which was a humbling experience to have happen. For our third playtest…this was a wow. With the feedback from testers, I feel optimistic about our odds of increasing the rating to 5.

Testers playing A Paycheck Away learning game

We had 17 testers and the overall “engagement” rating was 4+ on a 5-point scale. Here’s a few of the testers’ comments. While they gave us some ideas for what to change, here’s a smattering of the “wow” comments we heard:

  • It was completely engaging.
  • Homelessness – the real world. It was like being immersed in it, some of the same emotions in a safe way?
  • It will change the way you think.
  • It was pretty eye-opening. You can easily judge someone without knowing the situation. Lots of one step forward and two steps back without you having control over it.

So how did we get to “wow?” Games all have the common structural elements I’ve listed below. As we went through various iterations, we adjusted these elements based on testers’ feedback. I’m sharing some of what we adjusted so you can see how things changed as we went along. You can use these structural elements as a guide when designing your own games, taking notes on what to change as you play test:

  • Players – how many? Single player or multi-player? In A Paycheck Away we started with a 5-player game. Through testing, we adjusted to make one role a “game master” and allowed people to play individually or  in teams of two for the other roles.
  • Objectives – what’s the point of the game? What are players trying to do? In A Paycheck Away, the game objective is to get yourself – and as many fellow players as possible – out of homelessness within 3 months.
  • Procedures – How do you play? We had to specify what happens on a turn, how play proceeds, how the game ends. We adjusted procedures significantly through our testing, abandoning a week-by-week approach to a bi-monthly approach in the interest of time.
  • Rules – What’s allowed? What’s not? What consequences occur in response to specific actions? An example of a rule in our game is: If you have children under age 10 and you accept a job, you must pay $120/week per child for childcare. Some rules got abandoned and others got added as we tested with the focus being on reducing play complexity and maximizing “fun” of the experience.
  • Resources – what resources will players have and how will these be limited? In A Paycheck Away the resources – all of which are limited – include money, time, health, and jobs. Through play testing, we came up with the best combination of resources and limits.
  • Conflict – The game is no fun if there is no conflict built in…but we found that cooperation worked far better than competition in this game. We created conflict in other ways: by limiting resources or requiring players to decide between two tough choices.
  • Boundaries – players want to be immersed in the game, but they also need to recognize the boundaries of the game. We felt we struck a great balance. Players knew the game wasn’t real…but most reported strongly identifying with the character they played.
  • Outcome – this one is critical. For players to stay engaged, they must feel they have a realistic chance of hitting the game objective…but the outcome needs to be uncertain. Our testing revealed we did a great job with this one. It was a significant reason players reported a sense of flow while playing. They were fully engaged in the experience and vested in the outcome.

Next step? Take the show on the road and have a group of IUPUI students play the finished game and participate in a debrief session. We know the game works; now we need to figure out the optimal way to debrief the experience for participants. This debrief distinguishes our game from a game you might purchase in a game store. Players talk about their experience and what they learned from it…something that recreational games don’t deliberately set out to do.

Serious Games…One example of great design

I love games. I love to play games; I love to come up with games for others to play…and I have created several. What I have not yet done is to design more than a rudimentary online game…but I’m really eager to do so. One of the most fun things at last November’s DevLearn 2008 conference (sponsored by E-Learning Guild) was its focus on games. It inspired me and made me oh-so-anxious for the client who will come to me and say, “I want a learning game!”

I also took back a message from “The Serious Game Zone” (hosted by Dr. Alicia Sanchez who is a research scientist with a LOT of knowledge about games). Motivation is everything in a game…if the learner isn’t engaged, they will not play. Period. So…a game can have great instructional intent but if it is not fun, learners will not play. In a serious game, the “game” part of it is as critical as the “serious (aka learning)” part of it.

I spent time today playing one of the games I saw at Devlearn – and I am really impressed by its instructional design and its “fun” design.  “Moneytopia” is a  game that is designed to teach teens and twenty-somethings (though just about everyone in America could benefit, I believe) about handling and investing money…and living within their means. I kept wanting to play…and increase my financial portfolio.  To do that, I HAD to complete the short learning pieces (housed within my financial statment) centered on financial management because that was a key way that I grew my money. And…I couldn’t shortcut out of them because if I did then I missed the money-making opportunities embedded within them. And…on the instructional side…all of the learning activities they had me do were absolutely relevant to the content.  I didn’t feel like they were dumbing anything down – and each learning presentation was really short (two or three minutes).

Check out this game for yourselves. Allow yourself at least 15-20 minutes to play it to really get the flavor of the game and the instructional design that went into it. Let me know how you’d rate it on a “fun” scale and on an “learning” scale. I can immediately think of all sorts of ways to mimic the design for other applications – and I feel confident that the design of it is not too sophisticated for a lot of good instructional designers to imitate.  It appears to be a Flash-based game design, which means it is within the grasp of many of us.

Click the image to check out the game for yourself. Again, let me know what you think!

Moneytopia is a role-playing game that teaches you how to manage money.

Moneytopia is a role-playing game that teaches you how to manage money.