Emotion Technology SDK Internship EMPATH

Javascript Emotion SDK for YouTube, Facebook, and Twitch API


By: Jeffrey Lu, Software Engineer Intern

The new JavaScript SDK from Affectiva makes it easier than ever for developers to integrate emotion-sensing technology via your browser!

As an intern at Affectiva this summer, I was initially tasked with the job of updating a previous demo that was written using the SDK. This demo involved capturing your emotional reaction while playing a game for a short amount of time, and the code itself was unwieldy, consisting of logic to start the webcam, set up a web worker, and send and receive frames for processing. Using the SDK, however, I was pleasantly surprised to find all this logic wrapped up in several lines of code:

// get the div where the face video will appear
var face_video = document.getElementById("face-video-div");
// initialize the CameraDetector
var detector = new affdex.CameraDetector(face_video);
// choose emotions to detect
// start the detector
// add callback for when the detector is initialized
detector.addEventListener("onInitializeSuccess", function() {...});
// add callback for when a frame finishes processing
detector.addEventListener("onImageResultsSuccess", function(faces, image, timestamp) {...});

I came to see the “emotions detector” as a black box: pass in a frame for processing and get back emotion data for that frame.

Armed with the SDK, I started looking for cool ways to integrate it. After hearing about the application of Affectiva’s software to ad testing, I became interested in YouTube. Why not track your emotional reaction to any video you watch, not just ads? YouTube seemed like the closest thing to a universal video repository, so I started looking into their player and data APIs. Soon, I was able to implement a search feature and dynamically control a YouTube player, even in the face of poor Internet connection and buffering issues. I borrowed the UI from the game demo, rendering the emotions chart using d3. You can check out the completed demo on Github here (or at the link provided below).


Screenshot of YouTube demo with all emotions displayed on graph.

Naturally, the next step was to use the SDK with a live video or stream such as Twitch or YouTube Live. For both of these platforms, I found endpoints to read the top current streams, and for each stream I was able to get both the corresponding video and a chat embed. I searched around for examples of dynamic charts and found that CanvasJS worked quite nicely. By discarding a fraction of the data once the emotion lines reached the end of the chart, I achieved a scrolling effect that matched the real-time nature of the media itself.



Screenshot of Twitch demo with all emotions displayed on graph.

Beyond its emotion-analysis capabilities, Affectiva’s SDK can also match your facial expression with an emoji. I recreated Facebook’s floating emojis for Live Videos, but instead of taps or clicks, your facial expressions generate different emojis. Using Paper.js and anawesome library of emojis I found online, I was able to overlay a stream of happy, smirking, sad, surprised, and angry emojis on a video. However, because there was no convenient endpoint to pull current Live Videos, I resorted to a previously recorded Live Video for my demo.



Screenshot of Facebook demo with joy and surprise emojis firing.

In addition to the APIs and demos I mentioned above, I created a fun way to search Giphy by emotion and tried filtering photos from my Facebook profile by emotion. I again used CanvasJS to render the “emoji wheel” in the Giphy demo.

So far I’ve only listed a handful of individual use cases, but there are endless possibilities of emotion-enabling applications in the web browser. One can imagine thousands of viewers watching the same stream and a server that aggregates and calculates the average sentiment of everyone watching. Emotions can enhance social media, social networking, games, and many other web-based activities. So what are you waiting for? Give the Javascript SDK a try and start developing your own emotion-enabled applications today!


  • Original game demo – Play a Tower Platformer game and watch your gameplay synchronized with your emotions. Technologies used: d3
  • YouTube demo – Search for and watch any YouTube video, then play back the video synchronized with your emotions. Technologies used: d3
  • Twitch demo – Pick a current Twitch stream and see your emotions graphed real-time. Technologies used: CanvasJS
  • YouTube Live demo – Pick a current YouTube Live stream and see your emotions graphed real-time. Technologies used: CanvasJS
  • Facebook demo – Watch a previously live Facebook video and see your emotions turn into floating emojis. Technologies used: Paper.js
  • Giphy demo – Mimic an emoji to search for corresponding GIFs. Technologies used: CanvasJS

Careers in emotion recognition technology

Emotion Technology SDK Internship EMPATH