BLOG

Emotion Technology SDK

Javascript Emotion SDK for YouTube, Facebook, and Twitch API

08.08.16

By: Jeffrey Lu, Software Engineer Intern

The new JavaScript SDK from Affectiva makes it easier than ever for developers to integrate emotion-sensing technology via your browser!

As an intern at Affectiva this summer, I was initially tasked with the job of updating a previous demo that was written using the SDK. This demo involved capturing your emotional reaction while playing a game for a short amount of time, and the code itself was unwieldy, consisting of logic to start the webcam, set up a web worker, and send and receive frames for processing. Using the SDK, however, I was pleasantly surprised to find all this logic wrapped up in several lines of code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// get the div where the face video will appear
var face_video = document.getElementById("face-video-div");
 
// initialize the CameraDetector
var detector = new affdex.CameraDetector(face_video);
 
// choose emotions to detect
detector.detectAllEmotions();
 
// start the detector
detector.start();
 
// add callback for when the detector is initialized
detector.addEventListener("onInitializeSuccess", function() {...});
 
// add callback for when a frame finishes processing
detector.addEventListener("onImageResultsSuccess", function(faces, image, timestamp) {...});

I came to see the “emotions detector” as a black box: pass in a frame for processing and get back emotion data for that frame.

Armed with the SDK, I started looking for cool ways to integrate it. After hearing about the application of Affectiva’s software to ad testing, I became interested in YouTube. Why not track your emotional reaction to any video you watch, not just ads? YouTube seemed like the closest thing to a universal video repository, so I started looking into their player and data APIs. Soon, I was able to implement a search feature and dynamically control a YouTube player, even in the face of poor Internet connection and buffering issues. I borrowed the UI from the game demo, rendering the emotions chart using d3. You can check out the completed demo on Github here (or at the link provided below).

Screen-Shot-2016-08-08-at-10.34.24-AM-300x144.png

Screenshot of YouTube demo with all emotions displayed on graph.

Naturally, the next step was to use the SDK with a live video or stream such as Twitch or YouTube Live. For both of these platforms, I found endpoints to read the top current streams, and for each stream I was able to get both the corresponding video and a chat embed. I searched around for examples of dynamic charts and found that CanvasJS worked quite nicely. By discarding a fraction of the data once the emotion lines reached the end of the chart, I achieved a scrolling effect that matched the real-time nature of the media itself.

 

Screen-Shot-2016-08-08-at-10.34.33-AM-300x144.png

Screenshot of Twitch demo with all emotions displayed on graph.

Beyond its emotion-analysis capabilities, Affectiva’s SDK can also match your facial expression with an emoji. I recreated Facebook’s floating emojis for Live Videos, but instead of taps or clicks, your facial expressions generate different emojis. Using Paper.js and anawesome library of emojis I found online, I was able to overlay a stream of happy, smirking, sad, surprised, and angry emojis on a video. However, because there was no convenient endpoint to pull current Live Videos, I resorted to a previously recorded Live Video for my demo.

 

Screen-Shot-2016-08-08-at-10.34.43-AM-300x170.png

Screenshot of Facebook demo with joy and surprise emojis firing.

In addition to the APIs and demos I mentioned above, I created a fun way to search Giphy by emotion and tried filtering photos from my Facebook profile by emotion. I again used CanvasJS to render the “emoji wheel” in the Giphy demo.

So far I’ve only listed a handful of individual use cases, but there are endless possibilities of emotion-enabling applications in the web browser. One can imagine thousands of viewers watching the same stream and a server that aggregates and calculates the average sentiment of everyone watching. Emotions can enhance social media, social networking, games, and many other web-based activities. So what are you waiting for? Give the Javascript SDK a try and start developing your own emotion-enabled applications today!

Demos:

  • Original game demo – Play a Tower Platformer game and watch your gameplay synchronized with your emotions. Technologies used: d3
  • YouTube demo – Search for and watch any YouTube video, then play back the video synchronized with your emotions. Technologies used: d3
  • Twitch demo – Pick a current Twitch stream and see your emotions graphed real-time. Technologies used: CanvasJS
  • YouTube Live demo – Pick a current YouTube Live stream and see your emotions graphed real-time. Technologies used: CanvasJS
  • Facebook demo – Watch a previously live Facebook video and see your emotions turn into floating emojis. Technologies used: Paper.js
  • Giphy demo – Mimic an emoji to search for corresponding GIFs. Technologies used: CanvasJS

download emotion SDK to emotion enable your project

Emotion Technology SDK