By: Affectiva Staff

Ohio State University professor Aleix M. Martinez’s research is the first academic substantiation of compound emotions, which have long been recognized intuitively, or identified explicitly by creative communities of illustrators, actors, and animators.

Examples in popular culture include Drawing People by Joumana Medlej (2012) and Making Comics by Scott McCloud (2006, illustrated below).

We should note that Martinez’s work is based on posed expressions of emotion, where participants were prompted to act out a certain expression. In contrast, Affectiva’s facial expression detectors are trained using face videos gathered in real-world environments. The difference between posed and spontaneous expressions is important since people rarely mimic posed expressions in actual scenarios.

BEYOND THE BASIC SIX EMOTIONS

Affectiva and others have long argued that the face communicates a much richer repertoire of emotions than merely six emotions. Like others, we took issue with research that dwelled on the Basic Six at the exclusion of other emotions. Ekman himself wrote about many other facial expressions of emotion, beyond the basic ones. Seeking to expand beyond the “Basic Six”, Affectiva’s co-founder and Chief Science Officer, Dr. Rana el Kaliouby, in 2005 wrote her award-winning Ph.D. dissertation at Cambridge University. Her work led to development of a computer to read affective and cognitive states—complex emotional states like interest, confusion and concentration that build on the basic emotions. Her work was inspired by research into autism indicating that a small set of basic emotions provided little to help autistic individuals interpret the faces and emotions of others. The Mindreading DVD, a DVD used in autism therapies and training, is a collection of over 400 emotions expressed via voice and the face. El Kaliouby’s work spurred much excitement and she went on to co-found Affectiva. To el Kaliouby and Affectiva, it is exciting to see more researchers shift from a focus on basic emotions.

COMPUTING COMPLEX EMOTIONS IS HARD WORK – BUT WORTH THE EFFORT

Early on, when Affectiva first began testing media content in a scalable manner, we quickly confirmed Dr. el Kaliouby’s premise—it became clear that six basic emotions did not adequately address the complex array of expressions we encountered. Moreover, the prototypic faces of the six basic emotions were seldom observed. One of our earliest undertakings to investigate this phenomenon was a crowdsourced study around disgust where viewers’ reactions were captured as they watched outlandishly grotesque videos. The responses were thus natural and spontaneous, not artificial or posed. Our goal was to train a computational model of spontaneous disgust expressions.

We found that much of the time smile and disgust curves go up in tandem—that our panel of respondents had not acted with the pure posed disgust response as we had anticipated. On further examination, we noticed the respondents exhibited a combination of expressions ranging from the classic opened mouth retching gesture paired with a wrinkled nose to a pronounced smile paired with a lowered brow, averted gaze and a hand to the face. Thus there were combinations of expressions that must be ANDed together to form a complete story, with each discrete emotion being accurately assessed. Armed with this knowledge, we hardened our computer vision algorithms—and veered once and for all from a strict adherence to the Basic Six.

REAL-WORLD TESTS, REAL-WORLD RESULTS

To illustrate the importance of detecting combinations of expressions, below, we expose online viewers to two retch-inducing videos. The first depicts a popular TV show survivalist who actually did eat a squirming, 5-inch-long, beetle larvae. Viewer responses ranged from combinations of disgust + joy (“Eeeewwww”) to disgust + surprise (“You ATE that thing??”) to a bit of Disgust + Sadness (“pain empathy”) for good measure.

Our hapless viewers were then subjected to a PSA where a man drank yellow sludge (that is, fat) from a glass to illustrate the caloric pitfalls of soda consumption. This media generally triggered variation on a typical disgust response paired at times with expressions associated with sadness or anger (including at times, a lip corner depressor and the lowered brow).

Though this particular video is quite extreme, it is illustrative of reactions creator’s seek—that is, to induce visceral reactions, oftentimes in a humorous context.

SIX IS NOT ENOUGH

The examples demonstrate that it is the simultaneous interplay among the various expressions that yields the true tapestry of facial expressions, to indicate the real reaction to content. Affectiva believes that restricting facial coding analysis to a single channel among a mere six emotions greatly diminishes the array of insights available to researchers. We instead undertook an enormously complex effort to make Affdex unique in its ability to capture a range of emotions by accurately assessing discrete expressions and by detecting nuanced combinations of expressions as el Kaliouby’s and Martinez’s works illustrate.