Is it Fun? Colin Johanson on How ArenaNet Measures Success

This article is copied verbatim from an official source. Accordingly, please do not correct any typos or other errors before verifying that they do not exist in the original article.

Is it Fun? Colin Johanson on How ArenaNet Measures Success

By Colin Johanson June 19th, 2012

To begin this blog post, let me pose a question: How do you measure the success of an MMO?

Historically, it's been easy to point to success with traditional MMOs: subscription numbers were the ultimate means a company used to measure how well a game was doing, and customers typically looked at those same numbers as well to gauge the success of the game. The number of concurrent users—how many players are online at a given time—was also important, but that number was usually hidden from users, since it typically painted a less rosy picture of a game's health when compared to the number of active subscriptions.

Now let me pose a second question: If the success of a subscription-based MMO is measured by the number of people paying a monthly fee, how does that impact game design decisions?

The answer can be found in the mechanics and choices made in subscription-based MMOs, which keep customers actively playing by chasing something in the game through processes that take as long as possible. In other words, designers of traditional MMOs create content systems that take more time to keep people playing longer. If this is your business motivation and model so you keep getting paid, it makes sense and is an incredibly smart thing to do, and you need to support it.

When your game systems are designed to achieve the prime motivation of a subscription-based MMO, you run the risk of sacrificing quality to get as much content in as possible to fill that time. You get leveling systems that take insane amounts of grind to gain a level, loot drop systems that require doing a dungeon with a tiny chance the item you want can drop at the end, raid systems that need huge numbers of people online simultaneously to organize and play, thousands of wash/repeat item-collection or kill-mob quests or dailies with flavor text support, the best stat gear requiring crazy amounts of time to earn, etc.

But what if your business model isn't based on a subscription? What if your content-design motivations aren't driven by the need to create mechanics that keep people playing as long as possible? When looking at content design for Guild Wars 2, we've tried to ask the question: What if the development of the game was based on...wait for it...fun?

If we chose fun as our main metric for tracking success, can we flip the core paradigm and make design decisions based on what we'd like to play as game players? Can we focus our time on making meaningful and impactful content, rather than filler content meant to draw out the experience? Can we make something so much fun you might want to play it multiple times because it's fun, rather than making you do it because the game says you have to? It's how we played games while growing up. I can't tell you how many times I played Quest for Glory; the game didn't give me 25 daily quests I needed to log in and do—I played it multiple times because it was fun!

So if your key metric for success of your game is fun, how do you make content that fits that goal, and how do you know if you're succeeding?

It's easy to tell if a subscription-based game is hitting its metric of success, you simply look at the number of subscriptions; fun is much harder to define. To accomplish this, we've had to fundamentally redefine our development process of content in Guild Wars 2 around this concept of fun, and it starts with asking a very simple question that surprisingly isn't asked that often in game development: "Are you having fun?"

This metric of success impacted a lot of our early content-related design decisions for Guild Wars 2. Some examples include:

• Fun impacts loot collection. The rarest items in the game are not more powerful than other items, so you don't need them to be the best. The rarest items have unique looks to help your character feel that sense of accomplishment, but it's not required to play the game. We don't need to make mandatory gear treadmills, we make all of it optional, so those who find it fun to chase this prestigious gear can do so, but those who don't are just as powerful and get to have fun too.
• Fun impacts decisions. Every time you finish a dungeon you get tokens you can trade in for reward items that you want, rather than having a small chance of getting it as a drop, because it's more fun to always get rewarded for finishing with something you want to have!
• Fun impacts development. Explorable dungeons have multiple paths you can take and random events. Because of this you don't feel like you need to play the same dungeon over and over again if you want to chase the prestigious rewards at the end, but can instead mix up that experience to keep it fresh and fun.
• Fun impacts customization. The event and personal story systems allow you to get a sense of customization from your characters. Playing through the game, each character can experience completely different content, and the world can always stay fresh and new in the pursuit of new story lines, and an ever-changing dynamic event world. It means going back to a place you've already been with a character can be fun, and it means making a new character on an entirely different personal story chain can be fun as well.
• Fun impacts gameplay. The pursuit of fun in content led us to make many gameplay decisions, including:Everyone who helps kill a creature gets experience and loot, so you're not competing with other players; everyone gets rewarded for events with karma they can spend to buy rewards they want, rather than get a random roll of stuff they might not want; content scales in difficulty, so if more people show up, there is still stuff for you to do; everyone is able to revive one another, so you view other players as assets that can help you achieve your goals, rather than people who might get in your way; everyone can harvest resource nodes and get the rewards in the world together, rather than racing other people to them who might steal it from you. All of these things are just more fun!

We didn't just ask "Is this fun?" in early development, though; we also asked this question constantly throughout our development process, and in a lot of different ways.

First—and this is one of the things that I love most about ArenaNet—we ask our QA team to ask this question when they test everything that goes into the game. When they play an event, they don't just file bugs, they write suggestions and ideas for how to make it better. They give their feedback on the experience: Did they enjoy it? How could it be improved? How many rampaging rabid raccoons could be added to this event to make it amazing? They send this feedback directly to the designer building the event, and talk and coordinate with them to help make it better. I've never heard of a game company where the QA team is so integrated into the development process, where they can enact and impact change on a daily basis in the game. They aren't just testers, they are developers who help make every part of the game better, and they do this by constantly asking the question, "Is this fun?"

Next, we ask this question of the company as a whole. We do what we call "All Calls" and "Small Calls," where the entire company, or subsets of the company, play through parts of the game and give their feedback, comments, and suggestions. This helps us refine the content, and the key question we ask during all of this is "Is it fun?" We then continue this process with our closed alpha testers, and let thousands of people play through the content we've developed and ask them to leave detailed feedback and suggestions on if the content they are playing is fun. Our content designers patrol the internal feedback forums constantly for comments on the content they build, and make changes based on the feedback they have received.

Finally, we expand this process to the largest possible audience, to our beta test with hundreds of thousands of players. To get feedback on fun from an audience this large, we need to ask the question in a way that's simple for them to answer, and easy for us to condense the feedback down into simple-to-look-at numbers we can then act from. To do this, we added surveys to the game that occur after you finish story steps, renown regions, events, and dungeons. Each of these asks players a few simple questions, but the most important question we always ask? "On a scale of 1 to 5, how much fun was what you just did?" From this, we print out giant reports of survey information, then meet as subteams and target the content that isn't scoring well on "the fun factor" before brainstorming, together, on how to make that content more fun and exciting.

If our model was subscription based, we might be spending all this time racing to add as much filler content as possible to keep players chasing the carrot. Instead, as content designers with the goal of creating fun, we get to spend this time refining our content and making it amazing. As designers, this is both liberating and refreshing in an industry in which developers rarely get time from publishers to actually polish their games. (High-five, NCsoft!)

So remember: when you're playing in any of the upcoming beta weekends and that little survey pops up, tell us what you really think. Those metrics truly help us guide our work, and help us get a sense of if you're truly having fun. At the end of the weekend, jump on the Beta Forums and leave your true honest feedback—if you loved something, tell us why, and if you didn't enjoy something, please let us know that, too. We really do read and listen to the feedback in the pursuit of making the most fun game we possibly can, because we know if you're having fun, the game (and our company) will be fine in the long run.

I hope you come out of reading this with a bit of insight into how our content development process is different based on the metric we've chosen to gauge the success of our content model in Guild Wars 2. It's a bit different, and at times we're flying by the seat of our pants—but most importantly? It's fun!

As a fun aside, as a result of allowing our QA team to be so involved in the development process, it helps us find people with amazing minds for design based on their feedback and suggestions. Many of our current content designers on GW2 originally started their careers in our QA department. Names of a few of these folks who may have built some of your favorite events, story steps, or dungeons include: Will "Willipedia" Fairfield, Devon "The Knife" Carver, Matt "Wolf" Wuerffel, Kim "The Truth" Kirsch, Ben "Zen" Kirsch, Mike "Z-Axis" Zadorojny, Leah "Giggles" Rivera, Theo "Thor" Nguyen, Anthony "Put a Buff on It" Ordon, Leif "Fly-By" Chappelle, and Jordan "Action" Anton. (Note: Some of these nicknames may have been invented for this article, but will now be used around the office...constantly. —Colin "Optimist Prime" Johanson. He didn't start in QA, but Eric "The Commissar of Common Sense" Flannum deserves a shout out for helping make all of this possible by building a design culture that lets us chase fun!)