By: Ashley McManus, Marketing Manager; featuring William Pheloung, one of the key developers of the SYNC game

Games are designed to take us on journeys and adventures - and the best games are able to appeal to us on an emotional level. While these designs are intended to connect with us at a deeper level to take us on an emotional journey, the majority of games today do not sense and adapt to a player’s emotions...but what if they could?

Our emotion-sensing and analytics technology is transforming the gaming industry, giving developers the tools to create more immersive games and provide gamers the unique ability to drive gameplay with their emotions. The SYNC (or “Since You Never Came”) game, an Affectiva SDK user, was designed around this concept. The game opens with the loss of the main character’s love, and players advance the narrative by syncing their facial expressions with the character in order to help her to accept what has happened so she can begin to move on. SYNC won the “Excellence in Innovative Narrative” award at RPI’s 2016 gamefest, and more recently, was also just accepted to the International Academic Meaningful Play Conference.  You can download the game at itch.io.

We interviewed one of SYNC’s developers, William Pheloung, on how the app works, the process of developing it, and what he thinks is the future of emotion-enabled applications.

What’s the pitch for your app? (Why should someone download it?)

You have lost the love of your life. Overwhelmed by grief, you sit by her tomb, flipping through the pages of the journal that chronicles your life together. So begins SYNC, an interactive visual novel designed in Unity using emotion recognition software from Affectiva.

You assume the role of Alex, grieving the recent death of her lover, Ivy, in a car crash. Unable to move on, Alex can’t help but futilely imagine what she might have done differently to prevent the fateful accident. Players advance the narrative by syncing their facial expressions with those of Alex. Eventually, Alex (and through her, the player) is faced with a choice: to accept her doomed relationship so she can begin to move on, or to reject her relationship and remain trapped in the past.

What does SYNC do and how does it work?

SYNC is a love story video game, and is controlled by players’ facial expressions. The idea is that making facial expressions to control flow of the story is much more natural than keyboard and mouse, and it is intended to be a more direct connection with the characters. It is a single player story based game.

The game is split into 3 different parts: minigames, sections of dialogue, and journal entry sequences - and each is played out differently. For example, there are mini games where you as the player are driving and must focus on the screen in order to prevent yourself from getting into a car crash. Every time you blink, it causes snowflakes which leads to poor driving, and if your car swerves too much you get into a car crash. Other games include interactive sequences, where by expressing emotion (such as smiling to represent happiness), you can cause a flower petal to reverse back up into the tree where it came from.

Where did you get the idea to build SYNC?

Originally another one of the main developers, Yihao Zhu, made an initial version of sync which was a tablet game all about keeping in sync with internal beat or rhythm to world. I was impressed that he really knew what he was doing and had such a great vision for the game. I also thought it would be great to make the game he wanted, plus make it successful. I really liked his idea, passion & vision for product, so I got involved and brought on a writer, designer, etc. to help us build it out further.

What was your process in building it?

As a team, we really struggled with design. Though we had decided very early on the story, deciding the mechanics was complete chaos down to the last minute. It's also important to mention how Affidex really inspired most of the game's mechanics. We were going to make a music game and completely pivoted once we saw the capabilities of the emotion technology toolset.

So we then came up with ideas as a team, prototyped out a few different things that may work, then we would revisit those prototype as team. If we liked a prototype, we then would talk about how to expand it, then add the design layer on top of that. So it was definitely a scrum, iterative process of working against features we like and getting feedback from users regularly.

What role does emotion and emotion technology play in the concept of your app?

Essentially the role of emotion for SYNC is control. Emotion drives the narrative, and we couldn’t have nailed this capability without the Affectiva emotion recognition SDK. We imagined a game where you can completely control it with your facial expressions. Gamers empathize with characters more when they are more engaged with that method of control - and in the case of SYNC, that method of control is your face.

How did you arrive at Affectiva's technology to help achieve your app's vision? How did the SDK help you?

In the early process of discovering what we wanted to achieve and how we wanted to implement the gaming features, we found Affectiva’s emotion recognition technology and basically dropped our whole initial concept to re-build it around the emotional intelligence capability component. We reached out, found the SDK generally easy to use, and liked the level of control it gave us.

Do you have any other advice for those looking to build apps of their own?

The biggest advice I have is to start out with small concept, iterate on that and get as much outside feedback as early on as possible. A strong centralized vision is great, but it isn’t always the best when it’s not distributed to the rest of your team. The best thing you can do is to mess around and get right into the hands of users to test.

About William
William Pheloung is currently studying Computer Science and Videogame Design at Rensselaer Polytechnic Institute. He graduates in December of 2016 and will be working on Xbox's performance tools team after graduation. William is a determined and hardworking individual with a passion for programming. When he is not making games, he’s running the show at his Fraternity or playing guitar. You can see his full portfolio here.