By Jim Deal, Affectiva Director of Market Development and Gaming

Emotions in gaming have a long history. At one time, communicating emotional content was far more difficult than it is today. Think Mario or Space Invaders. Graphics cards and CPU clock speeds limited developers in a big way. Sixteen years ago I was art lead on Combat Flight Simulator 2. I remember being called out by the then-head of Microsoft Games Studios, Ed Fries. He wanted our art team to communicate the real emotional impacts of World War II. Not just the thrill of victory at a smoke trailing enemy plane, but the real consequences of war. His point was simply that our subject matter, World War II, had tremendous emotional impact on millions of people’s lives. Why not use it to affect the people playing our game?

Art: Kasey Quevedo

This was sixteen years ago, and our user interface team was limited by memory and the CPUs of the day. We could “afford” a few still images and a highly compressed animatic or two. We created “Manga” (Japanese Comic book) styled images of the war; mothers crying for dead sons, medals being awarded and stories of both American and Japanese young adults leaving home and learning to fly combat aircraft. The methods were bare bones but worked, at least for one gamer.  From this “media” I got a heartfelt fan mail from a 15 year old who was captivated by the stories we told. From this response, I knew there was something to “emotion” in gaming.

Today, the non-interactive communication of emotion in a game is much more sophisticated. I saw a cut scene not long ago that featured a 3D model that looked, talked and moved like Kevin Spacey. For all intents and purposes I was watching a performance by Kevin Spacey - filtered through a high tech combination of scanners, Hollywood motion capture and even a film score. But, it was undeniably Kevin Spacey on the screen and with him all the emotionally charged drama that an actor of that caliber commands. With current budgets and Hollywood directors, game designers and cutscene directors have the ability to elicit any kind of heart pumping tear jerking emotions that a good performance can deliver. Quite a jump from a few “Manga” styled still images and animatics.

However, is this “Hollywood” approach enough for future interactive experiences? We all watch movies, games should be more.  Additionally, very few developers can even take advantage of the current trend in passive techniques due to massive costs.

The interactive approach

To look in a different direction altogether, below is a video of a quick game demo done in Unity using the Affectiva Unity Plugin that shows how “Emotion Mimicry” can work to make a character reflect the emotional state of a gamer:

Games like Nevermind (find it on steam) use emotion recognition to adjust their gameplay real-time to adapt to a user’s reactions. A survival horror/puzzle title, the team at Flying Mollusk does an excellent job of altering the gameplay depending how much fear or stress the player exhibits.

Other ideas surrounding this tech include an emotion meter for combatants in a multiplayer game. Imagine knowing that the adversary you just destroyed online is now angry at you. Or perhaps there is a spectator “crowd” meter on a streaming service during a tournament showing the reactions of the crowd. Maybe there is a spot in a game where the guard must be charmed by a smile instead of overwhelmed by force. One team is using a combination of this tech and AR glasses to create a game that teaches autistic kids to read the emotions of the people around them, and know when they are going too far. (Read more on this use case.)

I think that emotional interactions in video games have come a long way. And now there is a new way to approach them. Not every developer has the money to hire Kevin Spacey, but tech like the Affectiva Unity Plugin is relatively affordable (and available). There are many wide open opportunities for  ambitious developers who are willing to take a chance on something new.

The Affectiva Unity Plugin works on Mac, Windows and Android with the iOS version coming out soon. It’s easy to install into Unity and and requires only a basic phone camera. You can download it now here.