Guestblog by: Erin Reynolds, Founder, President, and Creative Director of Flying Mollusk

I’m the Creator and Creative Director of a video game called Nevermind – an immersive biofeedback-enhanced atmospheric adventure experience that ushers you into dark and surreal worlds within the minds of psychological trauma patients.

When played with (optional) biofeedback technology, Nevermind senses your fear – lashing out if you allow your feelings of anxiety to get the better of you. Can you brave the terrors within Nevermind… and yourself? Currently available on the popular gaming online store Steam, it is one of the first commercially available games to use a player’s physiological and emotional responses to impact gameplay in real time.

Nevermind biofeedback in action

Nevermind originally started as my graduate thesis project at the University of Southern California back in 2011. After extensive research and trial and error, it was determined at the time that the best biofeedback input mechanism for the game would be a consumer-grade heart rate monitor. From there, we developed an internal algorithm to interpret Heart Rate Variability (HRV) from the heart rate monitor – from which the game would be able to derive any potential feelings of fear or stress in the player.

HRV can be a fantastic physiological indicator of what is sometimes described as “psychological arousal” – how “amped up” a person is. In creating Nevermind, we use those signals as indications that the player may be stressed or scared. Psychological arousal, however, only tells part of the story. It cannot convey the valence of an emotion, such as whether said arousal is positive or negative (i.e. happy or sad, basically). Since Nevermindfalls into the horror aesthetic, we have traditionally assumed that, when the player exhibited signs of elevated psychological arousal, that it was with a negative valence (i.e. fear, stress, or anxiety).

As one could imagine, we were naturally thrilled to hear about Affectiva’s Emotion SDK – we would no longer have to make assumptions! Affectiva’s emotion recognition software cansense valence and we were delighted to have the opportunity to combine our physiological biofeedback input with the power of Affectiva’s technology for a much more refined and accurate picture of how the player might be feeling at any given moment.

However….that wasn’t all. We realized that Affectiva’s technology was powerful enough to be able to offer players the full Nevermind biofeedback experience… all by itself! This was crucial for us because, up until that point, players would have to own one of Nevermind’ssupported heart rate sensors in order to experience Nevermind’s biofeedback functionality in action. With Affectiva’s Emotion AI, players with almost any basic webcam could experience biofeedback in Nevermind based on emotion recognition alone (instead of our traditionally supported physiological recognition). This was huge.

WEARING YOUR HEART ON YOUR… FACE?

However, in order to support pure emotion recognition as a viable input for Nevermind, we had to find a way to translate the experience of playing Nevermind with a psychological biofeedback input into a similar-feeling experience when playing it with an emotional biofeedback input. In other words, what emotions should Nevermind look for (via the Unity Affdex plugin) that would allow the game to react to feelings of fear, stress, and anxiety in the same way it does when responding to HRV? Affectiva’s Unity plug-in provides an impressive set of emotion-based qualities (e.g. Happiness, Anger, Fear, etc.) as well as more nuanced facial states (e.g. eyebrow raise, brow furrow, etc.), so there were a number of fascinating options to explore.

At first, we smugly thought to ourselves, “This should be easy! We’ll set Nevermind to respond to Fear!” We ran a few playtests, and the game just didn’t feel the same as it did when played with a heart rate monitor. The game’s response to the input wasn’t quite right.

That was weird. So now what? Maybe negative valence in general?

Nope.

Huh.

Baffled, we ended up creating an internal tool that would allow us to bring in testers into the studio, play the game, and track both their HRV input AND all of the many possible inputs that Affectiva’s emotion recognition can record. All this data could then be exported into a spreadsheet and, from there, we were able to sort, color code, chart, and graph all of the data to see where there are any correlations between physiological input and emotion-based input.

SO, WHAT IS THE ESSENCE OF FEAR IN NEVERMIND?

After extensive data analysis and becoming intimately familiar with Excel’s advanced conditional formatting and filtering, we stumbled upon some very interesting findings that not only solved the problem at hand, but also led us to learn more about Nevermind and what many of our players feel while playing.

It turns out that when players’ psychological arousal levels are elevated while engaging withNevermind, it’s often not due to sensations of fear in the sense of them feeling any particular threat. Rather, we saw stronger correlations with emotions like anger or disgust and even heightened attention levels.

Although it seemed strange at first, it actually makes a lot of sense. We used to call Nevermind a horror game, but it really isn’t “horror” by most video game standards.Nevermind doesn’t have many jump scares. There are no zombies chasing after you with a machete. You rarely walk into gore-filled rooms.

Instead, Nevermind is much more of a twisted, creepy, haunting experience that presents disturbing psychological scenarios that seek to “get under your skin” rather than make you jump out of it. So, it would make sense that players find themselves undergoing sensations of anger, disgust, and elevated concentration rather than fear alone or simply negative valence in and of itself.

NEVERMIND IS LISTENING

In the end, we were able to create a combination of the emotional inputs via Affectiva’s Unity SDK that now allows Nevermind players to have a similar biofeedback experience with only a webcam as those who play it with a heart rate sensor. Granted, the ideal scenario is, of course, when Nevermind is played with both the emotion-based biofeedback (powered by Affectiva’s technology) and physiological-based biofeedback (via a heart rate sensor) together. With both, the game has access to the full picture of both psychological arousal and valence. Regardless, if players play with just one or the other, they can enjoy a consistent responsive experience as they brave the darkly surreal and ever creepy worlds withinNevermind.

Regardless, perhaps one of the most interesting take-aways is a reminder of just how complex the human response to any scenario is – one of the many reasons why I’m so excited that we, the creative technology industry, are finally now starting to scratch the surface of learning about how to work with emotion recognition and Emotion AI to create deep listening experiences like none we’ve ever encountered before.

Don’t miss Erin’s blog: Why We Need Emotionally Sensitive Games

Follow Erin on Twitter: @reynoldsphobia