Last week at GDC, King gave attendees a peek behind the curtain at the complex process behind how it defines the success of each Candy Crush Saga level.
Head of central insights Jan Wedekind and senior director of data science Xavier Guardiola started by noting that millions of players have passed the 10,000th level in Candy Crush Saga, and most of them have not paid a penny.
Guardiola said that at launch, Candy Crush Saga’s notoriously difficult level 65 was originally intended to be the final stage of the game. It was too hard, and as a result it became both the highest converting level in the game and the stage that has “has made the most revenue ever,” said Guardiola.
But it has also caused the most player churn, he said. It was one of the first levels King A/B tested after launch, the results of which naturally showed that increasing difficulty raised conversions short term. Over the long run, though, making the level easier retained players longer.
“Make a level easier and you will make less conversions, and that’s not ideal,” Guardiola continued. “But in exchange for that, you’re saving players from churning…and those players, like compound interest, will start growing and in subsequent levels they may start spending.”
This reflects King’s long-term view of level testing, Guardiola continued. He said puzzle game designers should “look at the impact and the ripple effects that change might have for the whole progression…and be prepared for long A/B tests.”
Wedekind then took the stage to talk through how King segments its player base, and gauges the difficulty and fun in each level. The first thing to consider is that players are not identical and experience each level differently, he said.
King has built a model based on this premise that judges player progression and churn that actually works across its portfolio of games, not just in Candy Crush Saga. “We battle-tested this in practice many, many times,” said Wedekind.
“When you talk about difficulty and fun, they are inevitably entangled,” he continued. “If you make the level harder, the probability that more people churn goes up. So how do you know that a level is good or not? Is it bad because many people churn or is it just that many people churn because it’s difficult?”
Wedekind said that King measures how fun a level is by using a metric based on a combination of a ‘time to abandon’ number and a ‘time to pass’ figure.
“’Time to abandon’ tells us something about intrinsic motivation and the engagement of a level…what you find is that hard levels can be fun. Just because they’re hard doesn’t mean they’re not fun, and inversely you can find easy levels which don’t have a lot of churn on them but they’re not much fun.”
“I think for anybody doing games, doing levels, whether they are casual puzzle levels or even triple-A, you really want to figure out a way to disentangle fun from difficulty, because you get down very weird paths if you don’t, and you won’t be able to really judge what is going on in your game.”
Next Guardiola tackled the question of how long a level should be, based on King data. “The longer the level is, the less likely it is to be fun,” he said. “Does this mean that we can’t create long, fun levels? No, but you need to be extra careful because the odds are against you.”
King learned that the maximum recommended duration of a level was dependent on level difficulty. Or, in short, if a level is really hard, it should also be really short.
Next, Guardiola talked through how players are profiled based on skill level, and how that affects level design. It measures players’ wins, losses and number of attempts in each level.
“You can reliably capture the skill of your players in probably months or even weeks sometimes,” said Guardiola, who said that King maps player skill over its level designs to get an idea of how most people will experience them.
King uses this skill metric combined with its other ‘time to abandon’ and ‘time to pass’ benchmarks to identify which levels effectively offer the most fun, or as Guardiola puts it, have the most “puzzliness”. This is separate from difficulty, he said. From there, King found that the level type players found least engaging were the harder, more skill-based (or “puzzly”) levels.
Wedekind went on to explain how all of these metrics combined allows King to identify the 100 least fun levels in the game, and set about fixing them.
“The result was a very significant uplift in engagement,” he said. “We were removing frustration from the game and players were keeping alive in the game for much longer. And of course, if you have more players playing even longer in the game, there are more options for them to play, to watch an ad, to monetize. It’s good for the business.”
Wedekind said it is continually rinsing and repeating this process to optimise the game, and not just in Candy Crush Saga. This model is used across King’s portfolio.
Wrapping up, Wedekind concluded that “crazy hard levels never pay off, at least in the long term”, and that “designing for averages is not a good idea”. Ultimately, he added, “retention always wins”.
