Bringing AR Emoji to Life

From presenting information to sharing stories online, multimedia has become the dominant tool of communication on smartphones over the past years. Case in point: the stunning pictures, inventive GIFs, and creative videos produced by Samsung’s new Galaxy S9/S9+’s camera capture emotions that are simply beyond words.

But the Galaxy S9/S9+ is taking it up a notch with an all-new variety of intelligent and personalized emojis, which means you may never want to send plain text messages again. AR Emoji is designed to bring your unique facial expressions and add more fun to the emoji experience. But how did this AR-based feature come to life? Get a look behind Samsung’s lab doors for a close-up view.

01. The Problem

Regular emojis add more color to the conversation and are a fun way to communicate, but we recognized there are a lot of times when the existing set of emojis just don’t cut it. These lack personality – one smile looks no different from another. So we started looking for ways to bring out more character and personality in emojis.

02. The Goal

We thought in an ideal world, you would be able to teleport yourself and share how you feel with others then and there. So we wanted to come up with a way to replicate that experience but with the simplest tools possible, having seen so many people use messaging apps to communicate even with people right next to them and rely heavily on emojis rather than text. The focus quickly became how to create an image of the user and deliver multiple facial expressions and emotions along with that.

03. The Idea

To accomplish this, we decided to leverage the Galaxy S9/S9+’s camera technology. We thought, if users could take a selfie and transform it into a custom avatar and emoji, it would add something unique to the fun of communicating. In addition, we wanted to find a way to track people’s actual expressions so that they would have at their fingertips emojis that actually felt like their own. It was also important that we give users the best tools to express themselves however they want to look.

04. The Solution

The first step toward this was to use augmented reality to generate an avatar. With facial recognition technology, the Galaxy S9/S9+’s camera starts by producing an initial 3D rendering, using a data-based machine learning algorithm. Next, the camera analyzes the 2D image of the user and pulls information from its deep learning database to create the user’s facial features on the final 3D AR Emoji. To bring people’s unique expressions to life, we decided to include Face Tracking Technology, which tracks over 100 facial features and allows us to capture individual emotions on the spot.

In addition to this, Blend Shape technology helps deliver an even greater range of different facial expressions. Based on 75 basic facial expressions, Blend Shape allows us to reflect a wider set of user emotions by blending multiple expressions together. This is also the technology that has made our Disney emoji offerings possible. Another major step we took was to go through numerous sampling tests. From our labs, we conducted at least 100 tests on a weekly basis to make sure the new feature included enough data to appeal to users regardless of their tastes.

05. The Result

The end product of this is AR Emoji, which among other features offers a realistic or cartoon-like option for creating your own personalized My Emoji. These can be transformed into a set of 18 different expressions – gradually to expand to 54 – which were carefully created as universal visual representations of human emotions like joy, anger, sorrow, and pleasure. We wanted to offer emojis that would be able to deliver the same emotions across the world and went through a lot of research to understand how certain emotions are categorized and expressed on a fundamental level. In addition to the variety of emotions, AR Emoji also comes with a lot of versatility, allowing you to create the 3D avatars, either as a GIF, video or PNG file. Through updates and other developments, we’re continually looking for ways to help you simply be you. Check out some of the latest changes we made on April 23* and keep an eye out for additional improvements throughout the year that will include more hairstyles and other options.

06. A Simple How-to Guide

*Software update availability, functionality, features and schedule may differ by each market and its conditions, and are subject to change without notice or obligation.