The ‘emoji’ originated in 1997 as a simple pictogram used to express emotion. But after years of evolution, the technology has progressed to the point that the Galaxy S10 AR (augmented reality) Emoji can transform your face into an animated version of itself before your eyes. Once you have your avatar, your AR Emoji can mirror a wide range of emotions and body movements with advanced facial and body tracking. You can even impose your AR Emoji on top of a photo of yourself and watch it move just like you!

Samsung Newsroom met with the developers of the Galaxy S10 AR Emoji, Yong-Il Kim, HanJun Ryu, MinKyung Lee, Jinho Lim, and MinSheok Choi, to discuss how they are leading the future of AR Emoji, and what’s next for the technology.

Q. What is the current trajectory of AR Emoji development?

The AR Emoji feature is a way to express and represent yourself. As we switched over from the Galaxy S9 to the Galaxy S10, we changed our focus from the representative ‘Avatar’ to the ‘Stylized Avatar’, which portrays users’ individual traits so attractively that they can’t help but show it off! To achieve this, we have created avatars that users feel affectionate toward because they represent their individual traits, but which also maintain a universal aesthetic appeal.

People use AR Emoji to reproduce themselves in the online sphere. Therefore, the feature must be able to portray not only their physical appearance, but also their online personality. We are continually coming up with ways for users to more freely and uniquely personalize their AR Emojis.

Q. What makes AR Emoji special?

The AR Emoji feature’s biggest differentiator is that it starts off by making a complete, detailed figure that best reflects the user’s actual traits. AR Emoji has avatar shapes to suit everyone and features a huge number of ways for users to represent their unique characteristics.

The features available in AR Emoji on the Galaxy S10 line include ‘Mask’ mode, ‘Mini motion’ and ‘Live figure’, all of which can be used in a range of situations. Because the AR Emoji feature is a specialized camera function, it can be used within the default camera app. It has proven to be a popular feature among users.

With ‘Mini motion’, body movements are captured so that the AR Emoji moves just like the user.

Q. What was your process for developing AR Emoji?

During planning it is important to identify the users’ needs. To figure out what these were, we monitored our acquaintances, families and a range of online communities to make sure that we made an AR Emoji that everyone would be satisfied with.

In the design stage, we considered how we could best express users’ emotions through facial expressions and stickers. The most common human emotions were categorized, then we had to establish the various degrees of each emotion. For example, there are many different kinds of sadness. Some make you want to scream, some make you want burst into tears, and some only cause a downturn of the mouth. When we struggled to find specific references for certain emotions, our team stepped in as actors to record their emotions. During this process, having emotive team members was a big help! Our recording sessions were always full of laughter.

Also, as an AR Emoji is a full body avatar, we needed to include a wide range of fashionable clothing items, shoes and accessories to make sure users had enough options to choose from. Research was very important at this stage, and some of our team members scoured shopping malls to ensure that we chose the right clothing options.

The version of AR Emoji that’s available on the Galaxy S10 line has updated editing tools. In developing these, we had to make internal decisions about how we were going to structuralize and define every component of the 3D data. For instance, we had to make decisions like ‘let’s divide the bone structure of the arm into five parts, and cover two of them with the sleeve while the rest are just bare arm.’ This was a useful process though, because as we were defining the details, a number of other potential features occurred to us. For example, we added the shadows on the avatar’s backs during this stage.

For more customization and creativity, users can utilize their own AR Emoji to create stickers.

Q. Do you have any tips for how best to use the AR Emoji feature?

You can use your avatar to make stickers that can be sent via messenger. This feature allows you to send customized emoji stickers with any expression you like. You can also create memes or stickers that include text to give the image more context.

A mask mode that applies your avatar’s face over your own is another useful function. When you need to take a photo but you haven’t had time to get yourself camera-ready, you can cover your face with an emoji. AR Emojis can also be very helpful for the many online creators who like to dress up or disguise themselves while recording their videos.

‘Mini motion’ is a function that both adults and children can enjoy. If you enjoy dancing, it can be fun to have a virtual figure to dance along with, instead of dancing alone.

Users can have more fun with the ‘Mask mode’, which adds the AR Emoji’s face onto users’ faces.

Q. Can you tell us what’s currently being worked on with regards to AR Emoji?

We are thinking about how AR Emojis can be used in new ways, such as users being able to play as their avatars in games. We are also working on incorporating popular characters into AR Emoji, as well as allowing users to custom-add things like make-up and tattoos so they can better express themselves with their emojis.

The global trends are centered around incorporating more and more decorative tools. However, we have also tried to be focus on being inclusive, with features like artificial hands currently available, and artificial legs to become available soon. By continuing to add more customization tools to AR Emoji, we hope to help people better express themselves. But we also want to see AR Emojis become a place where users’ imaginations can be brought to life.

We’re looking for ways to allow users to represent not just their faces, but their bodies too. Another area of focus is finding a way to utilize AR Emojis on other Samsung apps. Users can currently set their phones to display contacts’ emojis when they call, but we are exploring other uses that include a mask mode for video calls too.

As developers, our priority is to improve usability. Currently, in ‘Live figure’ mode, users can tap the figure on the screen to trigger a range of gestures and dances. We want to make these motions more fluid, while also expanding the feature by incorporating further actions and promoting deeper interaction between users and their emojis.

Q. What does the future hold for AR Emoji?

I hope AR Emoji will offer users a chance to familiarize themselves with AR technology and become more comfortable using it. As virtual reality-based services are commercialized and become more common, we expect AR emojis to be the best method for people to represent themselves in the virtual sphere.

Ultimately, our goal is to make AR Emoji a personified visual assistant in the coming age of AI (Artificial Intelligence). For example, currently Bixby can give us updates on the weather, but I believe that information could be delivered more effectively by an AR Emoji holding an umbrella.

(Clockwise from the top left) Samsung developers Yong-Il Kim and HanJun Ryu from Unique Sales Point Strategy Group, MinKyung Lee from Media UX Group, Jinho Lim from Graphic R&D Group and Minsheok Choi from Visual S/W R&D Group

1. Jump to Your Favorite Apps Securely with Lock Screen ‘App Shortcuts’

Security meets convenience in the Galaxy S10 line’s lock screen app shortcuts. With this enhanced feature you can secure the apps pinned to your lock screen with a handy ‘finger scan and swipe’ motion. Simply scan your fingerprint as usual, then swipe up from the fingerprint sensor to an app icon to open it. In situations where you need to access your most-used apps quickly, snap straight to them from the lock screen without compromising on security.

The Galaxy S10’s in-display Ultrasonic Fingerprint Scanner offers vault-like security with its FIDO Alliance-certified anti-spoofing technology. Now, your fingerprint scanner is not only a top-notch security measure, but a platform to launch your favorite apps quickly and securely.

2. Streamline Your Life with ‘Bixby Routines’

Imagine; as soon as you get into your car in the morning, music fills the space and Bixby turns on voice controls, all without you having to touch a thing. Then, as you arrive at work, vibration mode is activated and Wi-Fi disconnects. As your day goes on, your smart assistant moves seamlessly through various other pre-arranged routines, ensuring that your device is always the perfect complement to where you are and what you’re doing. What’s more, once you’ve set a few routines manually, Bixby learns from your habits and suggests new routines that it thinks will suit you. There’s no need to be constantly taking your phone out to fiddle with settings when you’ve got ‘Bixby Routines’ to take care of the little things.

Setting up manual routines is easy too. If you’re taking a walk, set a routine so that music plays as soon as you connect your headphones, then have your phone activate Wi-Fi and open YouTube as you arrive back home. Playing a game? Set a routine so that you enter ‘Do not disturb’ mode upon opening the game, and notifications don’t distract you.

The list of conditions that can be programmed to trigger settings adjustments and new actions is endless. And with ‘Bixby Routines’, the more routines you set, the easier life becomes.

3. Keep a Hand Free with ‘One-handed Mode’

Have you ever been holding something important in one hand, and trying to use your phone with the other? How many of us have been in the middle of cooking, and when we go to check the recipe on our phones only one of our hands is clean? With the Galaxy S10 line’s edge-to-edge screen, reaching up to pull down the notifications bar or interact with other features near the top of the screen can require regripping the phone or using your other hand. But no more, with the Galaxy S10’s optimized ‘One-handed mode’.

This feature is easily activated with either a quick swipe up from the corner of the screen (with ‘Gesture Controls’ enabled), or a triple-tap of the home key. Once turned on, the feature brings up a miniature version of your screen that puts even the features right at the top of the display well within reach. After using one-handed mode to navigate to the feature you want to use, simply tap the black area next to the downsized screen to snap your display up to full size again.

4. Stretch Your Screen with ‘Gesture Controls’

Want to get the most out of your edge-to-edge AMOLED display? On the Galaxy S10 line, enabling ‘Full-screen gestures’ will remove the navigation bar at the bottom of your screen, and allow you to perform the same actions with simple swipes up from the left, right and center spots at the bottom of your screen. Prefer a different orientation for the swipe spots? Customize the layout and swap the positions for the ‘back’ and ‘recent apps’ swipe positions. The central swipe spot remains the ‘home’ option.

Smooth and intuitive, the Galaxy S10 line’s complete gesture control system allows for the removal of navigation buttons that take up your screen space. Three minimal grey lines are visible at the very bottom of the screen to avoid any confusion over swipe positions. Enabling gesture controls lets you take full advantage of your screen real estate and provides a glimpse into a truly button-less future.

5. Take a Smart Break with ‘Temporary Mute’

In a world where ‘digital detoxes’ are becoming commonplace, more and more users are making use of their devices’ ‘Do not disturb’ function. This feature mutes all calls, alerts and media (unless select exemptions are put into place) until the user chooses to turn it back off again.

But sometimes, users who have entered ‘Do not disturb’ mode when heading into a meeting or going to sleep risk missing important alerts if they forget to turn alerts back on. Enter, Samsung’s new ‘Temporary mute’ feature, which allows users to choose, down to the minute, how long their phone will remain on mute for. After they get out of their meetings or wake up in the morning, users’ devices automatically switch back to previous sound modes so that they don’t miss any important alerts.

6. Make Your Power Use ‘Smart’ with ‘Adaptive Power Saving’

With the Galaxy S10 line’s ‘Adaptive power saving’ mode, your phone will adjust battery settings throughout the day and maximize your battery life without you having to lift a finger. Based on Samsung’s AI technology, ‘Adaptive power saving’ studies your usage patterns and keeps frequently used apps running while preventing unused ones from consuming battery life in the background.

In high-usage periods where users are watching a lot of videos, playing games, or taking sudden trips that require them to use their phones a lot, the device will automatically enter power saving mode to extend power-on time. Users don’t always have time to monitor battery life and change their settings accordingly, but with ‘Adaptive power saving’ your phone has it covered!

7. Let ‘Shot Suggestion’ Guide You to Stunning Pictures

Utilizing machine learning gleaned from the analysis of more than 100 million photos, the Galaxy S10 line’s ‘Shot suggestion’ feature helps you take pictures like a pro. Once you turn the feature on, just follow the visual prompts on the screen and watch your ‘best shot’ fall effortlessly into place.

‘Shot suggestion’ is ideal for when you want to show a single person or place in a lot of different lights, as the AI suggests a varied range of compositional styles. Especially helpful for nature and landscape shots, ‘Shot suggestion’ ensures the horizon is level, while helping you improve your proportions and positioning. With the AI assistant, finding the perfect composition for your photos is a snap!

8. Compose Visual Media like a Pro with ‘Grid Lines’

The key when composing a shot from a ‘bird’s eye’ angle – for instance when taking photos of food –is to ensure the camera is held completely level.

The ‘Grid lines’ feature on the Galaxy S10 line helps you produce well-balanced bird’s eye shots. After the feature is turned on in settings, a levelling guide will appear in the middle of the screen to make sure you’re holding the phone perfectly flat. Simply align the circle in the center of the screen with the guide, and voila! Even a beginner can take well-composed bird’s eye shots every time.

The Galaxy S10 line’s grid mode is a lifesaver for vloggers too, as it helps ensure that videos taken on a selfie stick, or similar peripheral, are steady, with the subject well-centered. Pair the Galaxy S10’s ‘3X3 Grid’ view with the camera’s other smart optimization features and capture media that will make people say ‘wow’!

9. Let AI Touch Up Your Pictures with the Photo Editor’s ‘Style’ Features

Did you know you can now let AI edit your photos for you? First applied on the Galaxy S10 line, the ‘Style’ feature uses GAN deep learning to intelligently analyze your photo and ensure that the subject is distinguished from the background, while employing various effects including ‘Color Pencil’, ‘Comic’, ‘Watercolor’ and ‘Pastel’. These vivid effects are utilized in different ways for every individual picture, and really make the image look as though it was originally created in whichever medium you select.

All of these features are included in the Galaxy S10’s default ‘Gallery’ app, allowing users to easily add effects to their photos and removing the need to install separate apps. Simply select a photo in your gallery, click on the pencil icon at the bottom of the screen and select ‘Style’ from the ‘more options’ menu in the top right of the screen. Distinct from preset filters, now AI deep learning can uniquely apply its effects to best match your picture.

10. Animate Yourself with ‘Mini Motion AR Emoji’

The Galaxy S10 line offers an enhanced ‘AR Emoji’ experience with its new ‘Mini Motion’ feature. The upgraded capability employs advanced motion tracking technology to allow users to capture full-body movements and see them replicated by their avatar in real-time. With enhanced character design and naturalized movements and expressions, now your avatar can do whatever you can!

With streamlined avatar content creation, the Galaxy S10 line’s ‘Mini Motion’ feature makes life easier for content creators, and can even provide a dance partner for the kids to boogie along with. With constantly evolving motion tracking technology and the ability to detect even the smallest facial changes, ‘Mini Motion’ offers an exciting glimpse into a whole new world of personalized motion tracking.