AI Energy Mode gives you energy saving benefits for your Samsung products. You will be able to access and follow all this evolution directly from your SmartThings application.

AI Energy Mode brings all kinds of positive and useful solutions to the home, reducing costs and making life easier at all times with its many functions, by understanding your routine and adapting to save energy intelligently.

Find more here: 

“When Samsung first launched Bixby as a voice assistant, it was part of a vision to create a human-to-machine interface that makes life easier and can advance over time with Samsung Galaxy innovation,” said YoungJip Kim, Executive Vice President and Head of Artificial Intelligence Team in Mobile eXperience Business, Samsung Electronics. “Today, we introduce updates to build a more intelligent interface that is proactive and adaptive, giving people greater control over their mobile experience.”

Giving People Greater Control Over Their Mobile Experience

The new Bixby updates bring several new features and improvements that allow people to customize their user experience further, including the recent availability of Bixby Text Call¹ in English. Now, English speakers can answer calls from anywhere by typing a message, which Bixby converts to audio and communicates to the caller directly on their behalf.

Users can also personalize their Bixby Text Call voice. Using the new Bixby Custom Voice Creator,² users can record different sentences for Bixby to analyze and create an AI generated copy of their voice and tone. Currently available in Korean, this generated voice is planned to be compatible with other Samsung apps beyond phone calls. And for those who want to customize their wake word, Bixby now supports changes to the Custom wake-up³ phrase within Bixby settings as well, adding a new option to create a personalized wake-up phrase.

From Intelligent Assistant to Intelligent Platform

Today, Bixby can provide more seamless continuity and better attune to user needs with the introduction of new scenarios,⁴ including playing music depending on the type of exercise in Samsung Health or saving schedules in Calendar. Now, Bixby can better understand intent and process follow-up requests by understanding context and associating words previously used in interactions. For example, Bixby users can first launch a workout on Samsung Health and then ask Bixby to play music⁵ that best suits that exercise by saying “Play music for this workout.”

Enhanced Language Recognition, Processing and Response

In addition to these great new features, Samsung also expanded Bixby’s on-device AI support.⁶ Users can run key commands entirely offline, such as setting a timer, taking a screenshot or turning on the flashlight. By integrating on-device AI with its native applications,⁷ Samsung was able to further expand its language offering⁸ and capabilities with advanced AI-based voice dictation. These updates to Bixby will be made possible by the powerful innovation found in Samsung Galaxy devices, including the industry-leading performance that enables on-device AI applications.

Availability

Bixby updates will be available for Samsung Galaxy users in February⁹ through a software update. For more information, please visit www.samsung.com/global/galaxy/apps/bixby.

¹ Bixby Text Call is available on selective models including Galaxy S23, S23+, S23 Ultra, Z Fold4 and Z Flip4. It works on phones with One UI 4.1.1 or above for Korean and phones with One UI 5.1 or above for English.

² Bixby Custom Voice Creator is available on selective models including Galaxy S23, S23+ and S23 Ultra. It works on phones with One UI 5.0 or above. Availability of functions may vary depending on the device model. The application is available for download via settings within Bixby or the Samsung Phone application (native call app).

³ The feature is available in Korean as of date and on selective models including Galaxy S23, S23+ and S23 Ultra. Exact timing and availability of the updates may vary by market and are subject to change.

⁴ The eight scenarios are all currently available in Korean. More languages to be supported.

⁵ Supported music applications include Melon, Bugs, Genie and FLO as of date.

⁶ Availability of the on-device AI support may vary by market and model.

⁷ Supported native applications include those apps that require on-device AI support only.

⁸ The on-device mode currently supports English, Spanish, French, Italian, German and Korean.

⁹ Exact timing and availability of the updates may vary by user and market.

Like so many great innovations, the latest UDC was a group effort — driven by Samsung’s experts and consumers’ valuable feedback. To learn more about the enhanced UDC for Galaxy Z Fold4, Samsung Newsroom spoke with the experts at Samsung Research,¹ who collaborated on the technology.

Read on to see what John Seok-Jun Lee, Head of Computational Camera Part at Samsung Research America (SRA), and Alok Shukla, Head of AI Vision Solutions Part at Samsung R&D Institute India-Bangalore (SRI-B), had to say about their collaboration behind bringing enhanced UDC technology.

Q: What is an Under Display Camera (UDC)?

Seok-Jun: An Under Display Camera (UDC) is an innovative camera solution that enables an uninterrupted viewing experience on a mobile device by hiding the camera under the display and dedicating the whole screen to users while applications are running. It not only requires hardware innovation by placing a camera under a display panel but also requires algorithm innovation for restoring image quality — one of the most complex image restoration problems.

Q: How does the UDC work?

Alok: In UDC, a front-facing camera sits underneath the display. The screen pixel density is reduced in this area, allowing the screen to display imagery unhindered — but at the same time, not block light from reaching the camera when it’s in use. As the camera is placed underneath the display, UDC can suffer from poor image quality caused by diffraction artifacts, which results in flare, saturation, blur and haze. Therefore, while the UDC brings a better display experience, it also affects camera image quality and other downstream vision tasks. These complex and diverse distortions make restoring UDC images extremely challenging.

Q: Why is the UDC an important technology when it comes to Galaxy Z Fold4? What value does it provide to customers?

Seok-Jun: Samsung’s foldables are designed to be new and different, combining smartphone and immersive viewing experiences into one mobile device via breakthrough display technology. When folded, it provides smartphone-like use, and when unfolded, the display provides a more immersive and productive experience. Our consumer research showed that Galaxy Z Fold series users prefer the Cover Screen camera for capturing photos and selfies and the immersive Main Display for more productive tasks like watching videos, playing games and working on documents. Therefore, we thought UDC technology would be the perfect fit to meet their needs for an uninterrupted screen while still being capable of tasks like opening video calls. Samsung has launched the first-ever foldable UDC solution in Galaxy Z Fold3 and enhanced its quality further in the Galaxy Z Fold4.

Alok: Galaxy Z Fold4 truly unfolds a world of immersive display. UDC technology is at the core of the pursuit of a true full-screen display. By combining the leading innovation in display technology with advanced AI techniques for image restoration, Galaxy Z Fold4 provides the perfect balance between screen quality and camera image quality.

Q: How has the camera’s image quality improved since Galaxy Z Fold3?

Alok: Samsung Galaxy Z Fold3 showcased the world’s first UDC on a foldable. After the market release of Galaxy Z Fold3, one of the key feedbacks from consumers was to make the UDC less visible. On the Galaxy Z Fold4, sub-pixel arrangement was optimized to further hide the UDC.

As mentioned previously, when the camera is placed underneath the display, this can result in poor image quality caused by diffraction artifacts, which cause flare, saturation, blur and haze.

That’s why, in Galaxy Z Fold4, we have leveraged the power of AI, backed with multi-frame processing, to develop a novel computational imaging pipeline for UDC image restoration.

SRI-B, along with Visual SW R&D Group at MX Business, focused on developing AI Image Restoration for photo capture and preview to compensate for optical diffraction in UDC. At the same time, SRA’s Mobile Processor Innovation (MPI) Lab improved the multi-frame computational photography algorithm. Galaxy Z Fold4’s UDC solves key technical challenges in image quality to bring a unique experience to end customers.

Q: Walk us through the journey of the SRA-SRI partnership project. Why did your two teams collaborate?

Seok-Jun: The SRA MPI Lab has been working with SRI-B throughout MPI’s entire history at Samsung, from 2014’s Galaxy Note4 camera commercialization all the way to 2022’s Galaxy Z Fold4 and Z Flip4. SRA MPI has been leading MX multi-frame processing (MFP) technology R&D for UDC, Night Mode, Expert RAW and other technology. SRI-B has contributed to all commercialized SRA MPI’s MFP by providing amazing camera software, image quality tuning and resources toward commercialization as well as AI camera R&D. Our 3-way collaboration with MX R&D group, SRA MPI and SRI-B has been a success — especially when the worldwide team used time zone differences effectively to complete milestones on time for such a complicated project. It’s always a pleasure to work with them and deliver Galaxy devices’ camera successes together to accomplish our mutual goal of being the world’s best smartphone camera experience.

Q: Please explain your job role and your team’s contributions to the UDC. How did you contribute to making the best possible user experience?

Alok: Samsung Galaxy Z Fold4 UDC was a complex R&D project, and SRI-B’s role was to think in terms of end-to-end for the best consumer experience. This required multi-domain expertise and deep collaboration among multiple teams. Experts from SRI-B, SRA-MPI Lab and MX Visual SW R&D Group worked seamlessly to realize an end-to-end UDC experience.

SRI-B also played a key role in utilizing a Neural Processing Unit (NPU) to execute the AI model, achieving real-time performance in previews and significantly faster capture processing time than in Samsung Galaxy Z Fold3 UDC. Camera previews, video recordings and video calling scenarios need real-time performance without losing any frames. AI-based UDC restoration accelerated with NPU for preview processing was applied for the first time in Samsung Galaxy Z Fold4. Although real-time processing was not required, as camera capture requires higher image quality than a preview, achieving better image quality with less processing time was desired for user experience.

Q: What were some challenges you faced while developing this technology, and how did you overcome them?

Seok-Jun: Low light transmittance and light diffraction are the main challenges for UDC camera technology. Still, one more challenge, which is different from a punch hole camera image pipeline, is that UDC capture quality is affected by not only the camera module but also the display panel. SRA MPI’s AI-based 3D ISP algorithm was designed with the needed flexibility to tune the output image quality with the different display characteristics. Having such flexibility in 3D ISP helped to overcome such challenges.

Alok: SRI-B’s key focus was developing an AI image restoration module for Samsung Galaxy Z Fold4 UDC. These restorations are required for photo capture mode as well as video mode. Video recording and video calling scenarios have strict requirements for real-time processing within the constraints of current consumption.

AI-based UDC restoration was applied for the first time in Samsung Galaxy Z Fold4 UDC for video. To achieve a better video quality experience than Galaxy Z Fold3, we had to make the AI model lighter while still achieving good restoration quality. Multiple experiments were conducted in network design and training to achieve good quality in real-time. SRI-B extensively utilized NPU and advanced quantization techniques to prepare a lighter, highly optimized AI model that achieves real-time performance with low current consumption.

Q: What is your vision for future UDC iterations at Samsung?

Alok: Samsung Galaxy Z Fold4’s UDC has opened up a new dimension to the smartphone camera form factor. UDCs in the future will enable placing larger camera sensors and multiple cameras at flexible positions without compromising the viewing experience. Another big challenge is towards mass adoption of UDC technology in more products as the main front camera. For this to happen, there will need to be a significant improvement in camera image quality, processing time and display experience.

Seok-Jun: Since the Galaxy Z Fold4 UDC provides a much better experience in terms of hiding the camera under the display and improving image quality, the R&D direction for the upcoming flagship devices will work towards making the UDC completely invisible. This will provide the best uninterrupted immersive mobile display experience while achieving the same image quality as a punch-hole camera.

¹ Samsung Research, acting as Samsung Electronics’ advanced R&D hub, leads the development of future technologies for the company’s Device eXperience (DX) Division.

▲ (From left to right) Engineers Younghoon Jeong, Younghun Jo at Hardware Platform Lab, Sangmin Lee, David Jung at Picture Quality Solution Lab of Samsung’s Visual Display Business

Quantum Matrix Technology Pro Delivers Ultra-Fine Contrast With Details in Darkest Blacks and Lightest Whites

Samsung has continued to pave the way forward in innovation for picture quality by applying cutting-edge technologies to its latest TVs. The newly released Neo QLED 8K is an upgraded version of its previous QLED TV and comes with a wide range of innovative features. At the center of this advancement is Quantum Matrix Technology Pro, which takes advantage of the enhanced 14-bit gradation for greater control of the miniaturized light source — the Quantum Mini LEDs. This allows the TV to control its lighting across an astounding 16,384 steps, which is four times greater than conventional 12-bit displays at 4,096 steps, providing true-to-life colors previously unseen.

Shape Adaptive Light and Real Depth Enhancer Add Depth for Vivid, Lifelike Images

Newly introduced technologies, namely Shape Adaptive Light and Real Depth Enhancer, have taken the picture quality offered by Samsung TVs to new heights. These new features optimize contrast and produce clean and punchy images, adding a more realistic sense of depth to your screen.

Real Depth Enhancer improves picture quality by separating an object from a background. “Every image has an object and background with different requirements to bring out the optimal picture quality,” said Jo. “Real Depth Enhancer separates the object from the background to enhance the contrast of the object, making it look clear. In doing so, we can clearly distinguish the object from the background, creating a greater sense of depth and making the images pop.”

Shape Adaptive Light analyzes a scene and estimates the difference in the light layer by each section of a scene to control the light source. Shape Adaptive Light creates the effect of synergy with the latest Quantum Mini LED to precisely control and optimize the light source for an accurate contrast. “Simply put, it makes bright areas stay bright and the dark spaces around it stay dark. Additionally, the technology enables optimal contrast levels and allows details in the inkiest blacks, producing life-like picture quality,” said Jo.

Upgraded Neural Quantum Processor Allows Customized AI Upscaling for Any Given Scene

Technological prowess has allowed us to enjoy 8K picture quality from the comfort of one’s home. Although not all shows and movies are made in 8K, users are able to take full advantage of the latest TV tech Samsung has to offer. In fact, the real value of 8K TVs comes from Samsung’s AI upscaling technology, which offers next-level picture quality by analyzing the image in real-time and upgrading it to higher picture quality or even 8K, regardless of the original image quality. Neo QLED 8K raised the number of neural networks from 16 to 20 to strengthen the capability for image analysis and upscaling. Multi-intelligent AI networks from the Neural Quantum Processor improve not only images, but also cloud games in 8K resolution for seamless and natural gameplay.

“As the number of networks has gone up, we were able to upscale a wider range of content to deliver a better viewing experience for users,” said Jeong. “Samsung added neural networks to make the image of cloud games vivid and reduce noise. The neural networks immediately analyze a picture, optimize a network and upscale it to 8K so that users can enjoy a cleaner game image.”

Jeong shared Samsung’s ambition to provide a better viewing experience: “Samsung’s multi-intelligence AI network is set up to continuously expand, delivering optimal picture quality technology according to a viewing environment and the latest content trends.”

Samsung’s 2022 QLED TVs, the World’s First HDR Displays With ‘Pantone Validated’ Approval

Color is an important factor in evaluating a TV’s picture quality. Samsung has achieved 100% color volume since 2017 and displayed accurate and vivid colors. Samsung’s all-new 2022 QLED models, including Neo QLED 8K, became the world’s first HDR displays with ‘Pantone Validated’ certification. Pantone specializes in color and provides a global color standard, PMS (Pantone Matching System). The certification means that Samsung’s ability to accurately express colors is officially recognized. Neo QLED 8K was acknowledged for its accurate expression of 2,161 Pantone colors and 110 skin tone shades.

“Pantone Validated certification suggests that the color expression of Neo QLED 8K is almost on par with the colors we are able to detect with our eyes,” said Lee. “Technological advancements of Neo Quantum Processor, along with Quantum Dot material help our TVs accurately express more than 2,000 colors recognized by Pantone and also 110 skin tones, the most complicated and delicate colors.”

Smart Calibration — Automatically Calibrate and Optimize Picture Quality in an Easy and Convenient way

Samsung has been providing industry-leading picture quality through technical innovation. A classic example is Smart Calibration, which is embedded in the latest TV models in 2022.

Smart Calibration enables anyone with a Samsung Neo QLED to calibrate their screen by simply using their smartphone.² This allows users to calibrate their TVs for superb image quality through a simple step-by-step process with the Smart Calibration feature within the Smart Things app. All users have to do is open the Smart Calibration feature within the SmartThings app and point their smartphone camera towards the TV, the SmartThings app will then simplify the process to optimize picture quality. Users can choose between ‘Basic Mode’ and ‘Professional Mode’ according to how accurately they want to adjust picture quality. Basic Mode offers a quick calibration within 15 seconds, while Professional Mode optimizes more details of the picture and takes only 5-8 minutes to complete.

“Picture calibration normally requires expensive equipments, such as a measuring instrument and pattern generator. Smart Calibration enables users to adjust picture quality easily and conveniently with just their smartphone. Samsung developed this solution to help everyone easily calibrate their screens and enjoy accurate images,” said Jung. “Professionals who work with colors like photographers tend to calibrate their devices quite frequently — once every three months or six months. While this isn’t required for ordinary uses, it will be great for users to calibrate their screens using Smart Calibration when they first purchase the TV or when moving the TV to a different location.”

Samsung Neo QLED 8K, the First TV To Receive ‘Eye Care’ Certificate by Top Global Certification Institutes

There is now more amazing content available via OTT services than ever before, and as a result, more and more people are spending their leisure time watching this content on TVs. Therefore, comfortable viewing experience is becoming an important factor influencing TV purchasing decisions.

The 2021 and 2022 Samsung QLED TVs, including Neo QLED 8K, won the ‘Eye Care’ certification from Verband Deutscher Elektrotechniker (VDE) in Germany, a test & certification association. This makes them among the first TVs to be awarded a certificate specific to eye care. VDE’s ‘Eye Care’ certification includes ‘Safety for Eyes’, ‘Gentle to the eyes’, flicker level,³ uniformity⁴ and color fidelity⁵ assessments. ‘Safety for Eyes’ certification is assigned to products with the emission levels of blue light, ultraviolet rays and infrared rays that fall into the ‘Exempt Group’ according to the classification of emission limits set by the International Electrotechnical Commission (IEC).

‘Gentle to the eyes’ certification proves that the amount of melatonin suppression from a device meets the International Commission on Illumination (CIE) standard.

“As TVs become more of an entertainment hub in the home, there’s an increased demand for screens with top-tier picture quality that minimize eye strain. A comfortable viewing experience is quickly becoming a key factor in purchasing decisions, in addition to picture quality and performance. Neo QLED 8K, with its ‘Eye Care’ certification, meets these criteria, offering more comfortable watching experience to users,” said Jung.

Additionally, the Eye Comfort mode on Neo QLED 8K adjusts the screen’s brightness. It limits blue light to provide a more pleasant viewing experience, all done automatically by the TV’s Neural Quantum Processor. “AI technology automatically adjusts a screen’s brightness and color temperature according to a viewing environment so that users can comfortably enjoy their favorite content on their TV,” said Lee. “After sunset, you will see significantly warmer colors and reduced brightness, while during the daytime, the screen will be brighter and more vivid.”

AI technology To Provide Perfect Viewing Experience With Minimal Effort From the Users’ Side

As displays get larger and the technology for enhancing picture quality improves, so too does the standard for good picture quality. “The ultimate goal of advancing our technology in picture quality is to enable our TVs to automatically adjust their own picture quality to its optimal setting, producing a lifelike image. We will continue to develop cutting-edge technology to offer true-to-life picture quality to our consumers,” said Lee.

“Our users’ overall viewing experience is just as important as accurate color and sharpness,” said Jo. This means it is essential to help users conveniently enjoy the best picture quality. “We consistently strive to create a viewing experience where users can simply enjoy content anywhere, in any environment. Samsung is committed to advancing AI technology to provide perfect picture quality with minimal effort from the users’ side.”

¹ According to data released by market research firm Omdia

² Requires Galaxy S, Note, Z Flip or Z Fold series models released after January 2019, or an iPhone that supports Face ID. Supported models may differ depending on the situation. For more information, please check the product manual at samsung.com/support.

³ According to International Electrotechnical Commission(IEC) standard

⁴ According to International Standard Organization(ISO) standard

⁵ According to Information Display Measurement Standard(IDMS) standard

Part one of this two-part series focused on Samsung’s latest home appliances and displays. Here, Junghoe Kim of Samsung Research’s Robot Center¹ and Hoseong Kang from Samsung’s Mobile eXperience Business offer exclusive insights on the company’s latest robots and the new Galaxy S21 FE.

Robotics: Reimagining Robots as Life Companions

▲ Junghoe Kim of Samsung Research’s Robot Center stands alongside the Samsung Bot i interactive robot.

Q: What is the theme for Samsung Research’s showcase at CES 2022?

The overall concept for Samsung Research at CES 2022 is “Digital, Meet Physical”. We’ve gone to great lengths to show how the personalized experiences that millennials and zoomers value can be seamlessly connected to the digital world.

To that end, the ‘house’ that we set up at CES was designed to resemble a digital world like a metaverse. Our AI ​​Avatar was created to identify users’ locations in the real world using UWB (Ultra Wide Band) location recognition technology, and interact with them through the nearest smart home product.

At this year’s CES, two types of companion robots will be introduced: the Samsung Bot i, an interactive robot, and the Samsung Bot Handy, a manipulation robot that can assist with household chores. They can help users by arranging a video conference for an urgent meeting, or by setting the table for a meal. Our CES 2022 booth demonstrates how AI and robots can come together to introduce innovative user experiences to the home of the future.

Q: What are some of the Samsung Bot i and Samsung Bot Handy’s most innovative features and technologies?

The Samsung Bot i is an interactive robot that utilizes balance control technology. It moves like a living being, making dynamic and life-like movements. When it’s next to the user, it moves alongside them and offers various forms of support. When it’s away from the user, it manages various tasks for them using its telepresence² function.

The Samsung Bot Handy is a robot that helps out with various household chores using manipulation³ technology. This enables the robot to recognize objects at home and skillfully pick them up. Whereas at last year’s CES, we showcased a technology that could pick up a single object based on learning, this year, we will introduce a scenario called “tableware handling”. This technology enables the robot to pick items up after recognizing multiple objects and analyzing the surrounding environment.

▲ (From left) The Samsung Bot Handy, an AI Avatar and the Samsung Bot i

Q: How would you summarize Samsung Research’s R&D philosophy when it comes to robots, and where is the company’s research heading?

Samsung Research’s Robot Center is conducting research in various areas under the vision, “Samsung Bot, Touching Your Daily Life.” We strive to make our robots companions that help users in their daily lives by closely interacting with them and making their lives happier and more convenient. To achieve this, we’re taking core robot technologies like mobility and manipulation a step further by creating platforms based on our extensive expertise in AI, and we’re developing various robots based on research in those areas.

The Galaxy S21 FE: A Smartphone for Fans, by Fans

▲ Mobile eXperience team member Hoseong Kang holds up the Galaxy S21 FE, the newest S series smartphone, designed with feedback from Galaxy fans

Q: Which features of the newly unveiled Galaxy S21 FE should people pay special attention to?

The goal of the Galaxy S21 FE (Fan Edition) is to offer the Galaxy S series experience to even more users. It was created by combining the most beloved features of the Galaxy S21. What makes the S21 FE so special is that it makes the S21 series’ flagship experience available to more people while offering the same iconic contour-cut design, outstanding AI-powered camera, Dynamic AMOLED 2X display and flagship performance.

The camera is a crucially important part of the smartphone. Millennials and zoomers (members of Gen Z) in particular make use of the front-facing camera in a range of different ways. That’s why we equipped the S21 FE with a high-resolution, 32MP front camera that lets you take high-quality photos anytime, anywhere.

Q: Galaxy FE devices are designed to allow more people to enjoy the core flagship features that Galaxy users love. How are users’ opinions gathered on functions, and what insights have you gained through that research?

When planning FE devices, we collect feedback on the Galaxy series from thousands of fans. Through that research, we’ve been able to identify which aspects of the Galaxy S series fans love the most.

We listen closely to their feedback and identify the common perceptions about Galaxy devices that transcend regional and cultural boundaries. One of the insights that has come up repeatedly in our research is that many fans associate Galaxy devices with the idea of constant innovation. This is something that Galaxy devices have earned by driving innovation in the mobile space for more than a decade. It’s also the core philosophy that the FE line, and other future products, must live up to.

▲ The Samsung Galaxy S21 FE empowers fans to express themselves with four unique new colors

Q: Why is Samsung continuously expanding the Galaxy S series lineup? What is the ultimate goal?

When it comes to innovating with the mobile experience, Samsung is at the forefront. We have always been one step ahead of market demand, and we’ve been able to use the Galaxy brand to bring users new and innovative mobile experiences. The Galaxy S21 FE is the latest product of our approach to innovation. With its unique colors and design, it offers users a broader range of choices, helping them express themselves.

¹ Samsung Electronics’ advanced R&D hub, which leads the development of future technologies for its Device eXperience division

² Refers to a virtual video conferencing system that makes participants feel as if they are in the same room.

³ Refers to the skillful manipulation of objects with hands. This includes picking objects up.

But Samsung’s cutting-edge new foldable solutions couldn’t have been realized without the hard work and collaborative efforts of Samsung’s global network of engineers. Among other accomplishments, these employees were able to incorporate advanced technologies like AI to enable drawing and writing on the Galaxy Z Fold3’s large screen, and to allow the device to convert handwriting in a range of languages into on-screen text with S Pen.

To find out more about just what went into the development of the innovative new S Pen, Newsroom spoke to Olga Radyvonenko from the Visual Intelligence Team at Samsung Research Ukraine.

Q: Can you briefly introduce the Ukrainian Research Institute and the kind of work that goes on there?

Artificial intelligence (AI) is one of the key technologies of our R&D center. Our lab is focused on bringing Samsung users human-centric AI technologies that make input and interaction easy and enjoyable.

Q: The S Pen has now been applied to the Galaxy Z Fold3, making it the first of its kind among foldable smartphones. What role did you play in this process?

The major contributions of our lab to the Z Fold3 were the development of on-device deep learning technologies for handwritten content recognition and the technology for reducing S Pen input latency that combines advances in graphics and AI.

Q: What is the S Pen Software Development Kit? What particular advantages does it bring?

The S Pen Software Development Kit (SDK) is being developed by Samsung researchers and developers from many countries around the world, and we are very proud to be part of this international team. The S Pen SDK provides ways to speed up rendering and to reduce and optimize S Pen input latency. In addition, it comprises smart AI-powered features, such as recognition of handwritten text, shapes, diagrams and handwriting straightening.

Q: Given the unique function of the S Pen on the Z Fold3, was there any function of the S Pen that was developed or expanded specifically for the foldable smartphone?

As the new Z Fold3 has a massive and unique foldable display, we’ve focused on using our AI technology to optimize the S Pen experience as well as improve the latency. Thanks to our point prediction technology powered by deep learning, we are able to predict the next movement of the S Pen to minimize the delay between input and output on the screen of the Z Fold3. This, in combination with the front buffer rendering technology and unique large screen, means that our users can enjoy the most natural handwriting, note-taking and drawing on their Z Fold3.

Q: How many languages does the S Pen now support? What were some of the difficulties of expanding the number of recognized languages for the Z Fold3’s S Pen?

At this moment, the S Pen supports 88 languages in total, 14 of which were newly added in the past year.

Adding a new language for handwriting recognition requires a lot of effort. In order to achieve the world’s best level of recognition quality and speed, we have developed an advanced AI technology. This technology is based on deep learning across millions of users’ samples and language modeling for natural language processing (NLP).

Q: Do you have any special tips for making the most out of the S Pen on the Z Fold3 to share with users?

For those who love writing by hand, the S Pen To Text feature now supports writing with the S Pen directly into text fields. The handwriting is then automatically converted to text. Also, you can edit any printed text with your S Pen – for example, when correcting spacing or inserting words.

One feature that will also interest many users is Neat Shape mode. It can help you to not only draw shapes, but also build a flowchart.

Make sure to check the Tips application and not skip usage tips – there you will discover the newest S Pen and input features, and also learn how to use many other Samsung apps to their fullest potential.

Q: Looking forward, what do you think the future of the S Pen looks like? How might it evolve?

We hope to introduce more ways to support better user experiences by providing even more personalization in the near future. Our mission is to make S Pen input like writing in your notepad with a pen, but better and smarter.

Most of all, I want for as many people as possible to experience the power and convenience of the S Pen and its handwritten input capabilities. It has the potential to enhance input on a wide range of devices. We view the S Pen not only as a cool gadget, but first and foremost as an accessible and useful tool. I hope I will see the S Pen and its features casually used in education, in the workplace and in everyday life. I do believe we are set to see a revolution in input technology.

The full interview video can be viewed here:

To help you get through your busy day, Samsung Electronics has put together a team. With the SmartThings app on the Galaxy Z Fold3 leading the way, Samsung’s latest washer and dryer, Neo QLED 8K and the Bespoke Jet Bot AI+ are ready to team up to help you get ready.

Read on to find out how Team Samsung helps its user ‘A’, breeze through his busy morning.

Good Morning Mode Opens the Curtains and Displays the Weather

A was watching his favorite movie last night and fell asleep late. He ended up missing the alarm on his smartphone, but there’s no need to worry. All he needs to do to get ready for the day is simply unfold his Galaxy Z Fold3 and activate Good Morning Mode in SmartThings.¹ Once he’s done that, the curtains that were closed so he could get a good night’s sleep automatically open up and the lights in his home switch on to help him begin his morning in a bright mood.²

One of the most important factors to consider when getting ready to go out is the weather. It’s the first thing to check before choosing your outfit and deciding whether or not to take an umbrella with you. Because Good Morning Mode displays the weather automatically through A’s Neo QLED, there’s no need to use the remote to check the weather.

A Washer That Chooses the Best Wash Cycle While Staying Eco-Conscious

A has picked out an outfit of the day that’s appropriate for the weather, but he forgot to start his laundry. Luckily, Team Samsung can select the best wash cycle and get his laundry done on time even when he’s not around.

Here’s how it works. First, open up the SmartThings app, tap on Laundry Recipe and choose the color, soil level, and types of clothes you want to wash in your washer. Wash cycles will differ depending on the color and soil level of the clothes, and whether you want to wash shirts or things like socks. The feature will automatically choose the best wash cycle based on your input.

The Laundry Planner feature allows you to specify when you’d like the load to finish, so your clothes will be fully dry and look good to go when you need them.

Whipping Up a Delicious Breakfast With SmartThings Cooking

A hearty breakfast is a must for starting off the day with lots of energy. Luckily for A, Team Samsung makes it easy to pick out a recipe for a great meal.

After receiving a recommendation on his Galaxy Z Fold3 using SmartThings Cooking, A automatically sends the recipe to the smart oven and starts cooking. When it’s all done, he can receive a notification through his Galaxy Watch, Family Hub, or TV.

Forgot to Turn Off Your Lights or Devices? Don’t Worry With Away Mode

After putting on some fresh, clean clothes and finishing a delicious breakfast, it’s finally time for A to head out the door. With SmartThings’ Away Mode,³ if A realizes that he forgot to turn off a light or the TV as he’s heading out the door, he won’t need to head back inside to do so. The feature allows you to control not just the lights and the TV, but devices like robotic vacuum cleaners (RVC) as well. In addition, if you have a pet who’ll be left alone at home, SmartThings Pet⁴ offers peace of mind by allowing you to monitor your furry friend.

Monitoring Energy Usage With SmartThings Energy

After Team Samsung helped him get ready and he headed out the door, while on his way to his destination, A became curious about how much energy he’d been consuming this month. Once he unfolded his Galaxy Z Fold3, he accessed the SmartThings Energy⁵ service, which offered him detailed insights on his energy usage.

For those who can’t afford to miss work or skip chores in this busy world, Team Samsung is here for you. Experience a smarter and more relaxing start to your day with SmartThings.

For more information on Team Samsung, please visit samsung.com.

¹ SmartThings application must be installed and connected to devices in order to utilize its features. Application requirements may differ depending on the device.

² The electric curtains and lights described are products of a partner company, not Samsung. Therefore, quality assurance depends on the partner company’s standards.

³ SmartThings application must be installed and connected to devices in order to utilize its features. Application requirements may differ depending on the device.

⁴ SmartThings Pet is available for Bespoke Jet Bot AI+.

⁵ How much detergent, water, and energy can be saved can differ depending on the actual usage environment and the types of items being washed. SmartThings Energy service may not be supported by some devices.

The global market for large screen televisions is on a surge. According to global market research firm Omdia, global shipments of TVs with displays over 75-inches are expected to increase by 17% compared to 2019 and 8K TVs are set to increase by 110% this year as well.

Given the market demands, Samsung is doubling down on its efforts to give consumers large screen TVs that not only offer the best picture quality but also the best sound experiences.

At the heart of that is Samsung’s AI Quantum Processor.

The AI Quantum Processor is a high-performing chip developed using Samsung’s proprietary technologies that can process complex algorithms quickly using machine learning-based upscaling. That gives consumers the ability to stream 8K content in real-time paired with dynamic, location-based acoustics.

Samsung is peeling back the secrets of the AI Quantum Processor to share on how it brings these experiences together to get better resolution, better sound and better picture integrity.

One of the leading questions consumers continue to ask about large screen TVs is how to watch ultra HD content in 8K resolution given that it isn’t widely available yet. That’s where Samsung’s AI upscaling technology comes in.

Today, the technology transforms not only 4K content into 8K, but it also works for lower-resolution SD, HD and FHD-quality content as well. The AI Quantum Processor 8K combines the advantages of existing machine learning capabilities with new deep learning technologies to analyze and deliver content at some of the most granular levels possible.

This hybrid approach helps to remove unnecessary noise and creates life-like textures and smooth edges on upscaled 8K content that only Samsung can deliver. It also adjusts content brightness and darkness levels according to the light and living room conditions the display is set up in.

AI Quantum Sound for the Most Suitable Audio Experiences

Adding a truly effective audio experience to high-quality picture quality serves to bring about an even deeper sense of immersion to viewers. Samsung’s AI Quantum Sound, bolstered by artificial intelligence, provides a listening experience similar to that of a movie theater right in your living room.

Samsung’s new innovative sound technology Object Tracking Sound+ follows movement on-screen to match the direction of the sound to the action. The technology recognized and analyzes the recorded audio synced to the content in real-time and evenly distributes the audio tracks to the six speakers embedded in the TV.

And due in part to the Active Voice Amplifier (AVA), users will never miss a word of dialogue, regardless of what is happening around the TV. The feature monitors the sound environment so that if there is noise louder than the TV’s audio, it will increase the volume of just the dialogue audio so as not to overwhelm users while ensuring no word goes unheard. Also, sound will be delivered seamlessly thanks to real-time deep learning audio separation technology, high-performance NPU (Neural Processing Unit).

AI Streaming for Minimal Data Loss

Thanks to the widespread use of the Over the Top (OTT) streaming services, more and more users are taking advantage of their large screen TVs and stream 8K content. However, existing network infrastructures cannot always guarantee image quality that meets users’ high-resolution expectations. This is why at the Samsung Developers’ Conference in 2019, Samsung introduced its AI Scalenet technology, designed to overcome these infrastructure limitations and support effective 8K streaming and low-latency livestreaming.

AI Scalenet is a CES 2020 Innovation Award-winning AI codec technology that generates resolutions four times higher than conventional technologies without any change to the network infrastructure. It minimizes the data loss that may occur during the compression and playing of videos through deep learning technologies, meaning users can enjoy 8K-quality content regardless of their network infrastructure. Even during peak usage, AI Scalenet dramatically improves image quality and content speed according to the network status and reduces picture breaking and ambient noise.

These days, more content than ever can be enjoyed on our TVs, meaning that the expectation for immersive viewing experiences in the home has increased, too. Samsung plans to continue to use a range of innovative technologies, starting with artificial intelligence, to bring the best experience 8K content and audio experiences to living rooms across the world.

(From left) Hyungjun Lim, Hyunseung Lee, Cheon Lee and Seungho Park of the Picture Quality Solution Lab of Samsung Electronics’ Visual Display Business

In recent times, screen picture quality has transcended high definition – these days, it’s all about ultra-high definition. Although today’s hardware has been developed to support not just 4K content but 8K content as well, only recently has content begun to actually be produced to 8K quality standards. Therefore, in order to improve the video display quality of content made before 8K existed, upscaling technology exists to keep up with today’s ultra-high definition standard.

Samsung Electronics’ 2020 QLED 8K TV lineup continuously receives praise for its devices’ innovative display quality, sound, design and many other areas. In particular, the range showcases an impressive evolution with regards to its video upscaling technology, at the core of which is Samsung’s AI Quantum Processor 8K.

Samsung Newsroom interviewed the team from the Picture Quality Solution Lab of Samsung Electronics’ Visual Display Business to learn more about this revolutionary technology.

Leveling Up Deep Learning for Optimized Upscaling

When a low-resolution video plays on a high-resolution display, the resulting image quality can be below a viewer’s expectations. AI (artificial intelligence) upscaling is a technology that converts low-resolution videos into ones of more vivid and natural resolutions by analyzing the video data and adapting it intelligently.

In 2018, Samsung Electronics implemented its machine learning-based AI 8K upscaling technology for the first time. This technology harnesses machine learning wherein a human-designed algorithm intelligently analyzes the various video qualities, including line, light and color, and improves each of these for an improved display resolution. Such deep learning capabilities permit a TV to analyze and process video features whilst simultaneously learning thanks to an artificial neural network that acts in the same way as the human brain.

The members of Samsung’s Picture Quality Solution Lab focused on developing deep learning algorithms that enable displays to always showcase optimum video qualities without human intervention. “Deep learning enables more precise and efficient image quality improvements than can be achieved by humans alone,” noted Hyungjun Lim, who developed the AI upscaling technology’s texture generation method.

This operational advantage is what led the team to develop an AI upscaling technology that combines machine learning with deep learning – Samsung’s AI Quantum Processor 8K. “Previous machine learning technology brought about enhanced sharpness to display picture quality, but now our technology can also offer more delicate texturing,” noted Hyunseung Lee, who developed enhanced edge sharpness for the AI Quantum Processor 8K. “Images with complicated textures, like mountain or grass landscapes, can now be upscaled into 8K quality more naturally.”

Solving the Conundrum of Deep Learning

Although deep learning has limitless potential, there are still obstacles to be overcome. The Picture Quality Solution Lab team had to go through countless trial-and-error scenarios in order to make their technology market-ready. “It was difficult for us, at first, to clearly follow and comprehend the algorithm being developed by the artificial neural network of the processor,” highlighted Cheon Lee, the developer of the AI Quantum Processor 8K’s network structure and learning method. “The artificial neural network’s high hardware chip power consumption was an issue we had to solve, too.”

After a lengthy development process, Samsung was eventually able to solve the problems that had arisen thanks to its range of differentiated technologies. “We achieved the highly efficient display quality performance of our processor by making it a hybrid technology that combines complexity along with different aspects of deep learning and machine learning,” explained Lee.

Dreaming Bigger, Better and More Realistic with AI        

It wasn’t that long ago that the day-to-day application of AI-powered technologies was but a distant dream to us. Nowadays, technology has improved rapidly to the end of improving the lives of people across many different areas of everyday life.

“We are now able to categorize videos of differing quality levels in a more detailed way while also bringing these AI categorization technologies to the commercial market,” noted Seungho Park, who worked the AI Quantum Processor 8K’s noise elimination and improving the detail contrast range. “Such technologies permit next-generation displays, such as The Wall, to be able to showcase all kinds of videos at the most realistic qualities possible today.”

This AI-powered innovation is not just limited to upscaling technologies. There exist a variety of ways to harness AI to improve the screen qualities of displays, including video quality categorization, noise elimination, data-compression-caused distortion restoration and spatial or time-based resolution restoration, to name just a few. “Our goal is to continue Samsung’s legacy of providing leading TV viewing experiences by combining the advantages of our next-generation displays with AI technologies that can restore videos to realistic-level qualities, regardless of their original qualities and resolutions,” affirmed Park. The future of QLED 8K is looking bright.

Samsung’s AI-enabled and smart home appliances can do just that for you, making short work of managing workout routines, cooking meals, driving in the city and even sleeping and getting up on time!

Samsung home appliances’ seamless connections are all part of the company’s “Intelligence of Things” vision, which envisions a world in which the devices around us are constantly communicating with one another in ways that simplify our day.

To demonstrate how Samsung smart home appliances aid you, check out a typical day in a connected home.

[7 AM]: Waking up with a soothing alarm

Setting an alarm on your Galaxy smartphone or Galaxy Watch is as easy as talking to
Bixby.

 [8 AM]: Work-outs are now more fun

Sometimes, taking time out from a busy day for your morning workout may seem too tedious. But with Bixby, you can easily map out your day through personalized services that are powered by natural interactions. You can ask Bixby on your phone to play the perfect music while you plug in your Galaxy Buds as you go for a run in the park. You can check the step count on your Galaxy Watch or on your Galaxy smartphone using the Samsung Health app.

[9: 30 AM]: An AI-powered recipe for an awesome breakfast

With Bixby at your service, everyday tasks such as preparing breakfast by checking what is available in the refrigerator is possible with a simple voice command. As you interact more frequently with your smart devices, Bixby learns usage patterns and frequently used apps and keeps getting smarter and faster.

[1 PM]: The Movie Playlist

A single house can contain multiple spaces that require different environments. Samsung’s AI service enables multiple modes where different devices can be clubbed to perform specific actions. So turn on the Movie Mode, and Bixby will automatically control the lights, air conditioning, TV, music…all in one go.  All that is left to do then is ask Bixby to play the movie of your choice.

[4 PM]: The perfect weekend drive

Plan a quick drive in the evening with Bixby. Samsung’s AI service can help you navigate to the local park or nearby places with just a simple command. Don’t forget to check the weather forecast on your watch before planning your outing.

[7.30 PM]: Getting your home ready for you

How about getting your home ready before your arrival? Simply ask Bixby to set the ambience just the way you like it. Turn on the Air Conditioner, switch on the lights and play some soothing music on your arrival!

[10 PM]: A peaceful sleep

Finally time to say good night. Control your smart devices with just your voice and they can work seamlessly. Bixby is always ready to listen and carry out your requests on devices that are connected through Smart Things.  From setting the alarm, to switching off all the lights in the master bedroom, it’s all just a voice command away!

[Giridhar Jakki, Senior Director, Voice Intelligence Team at Samsung R&D Institute, Bengaluru has contributed to this article. The Bixby Team comprising Product Managers, Designers and Developers are determined to bring valued experience to customers. Bixby’s understanding of Indian accent and localization activities in Mobile, TV, Family Hub and other appliances are majorly developed at the Bengaluru centre by collaborating with Suwon R&D center and other Indian R&D centers in Noida and Delhi]