But Samsung’s cutting-edge new foldable solutions couldn’t have been realized without the hard work and collaborative efforts of Samsung’s global network of engineers. Among other accomplishments, these employees were able to incorporate advanced technologies like AI to enable drawing and writing on the Galaxy Z Fold3’s large screen, and to allow the device to convert handwriting in a range of languages into on-screen text with S Pen.

To find out more about just what went into the development of the innovative new S Pen, Newsroom spoke to Olga Radyvonenko from the Visual Intelligence Team at Samsung Research Ukraine.

Q: Can you briefly introduce the Ukrainian Research Institute and the kind of work that goes on there?

Artificial intelligence (AI) is one of the key technologies of our R&D center. Our lab is focused on bringing Samsung users human-centric AI technologies that make input and interaction easy and enjoyable.

Q: The S Pen has now been applied to the Galaxy Z Fold3, making it the first of its kind among foldable smartphones. What role did you play in this process?

The major contributions of our lab to the Z Fold3 were the development of on-device deep learning technologies for handwritten content recognition and the technology for reducing S Pen input latency that combines advances in graphics and AI.

Q: What is the S Pen Software Development Kit? What particular advantages does it bring?

The S Pen Software Development Kit (SDK) is being developed by Samsung researchers and developers from many countries around the world, and we are very proud to be part of this international team. The S Pen SDK provides ways to speed up rendering and to reduce and optimize S Pen input latency. In addition, it comprises smart AI-powered features, such as recognition of handwritten text, shapes, diagrams and handwriting straightening.

Q: Given the unique function of the S Pen on the Z Fold3, was there any function of the S Pen that was developed or expanded specifically for the foldable smartphone?

As the new Z Fold3 has a massive and unique foldable display, we’ve focused on using our AI technology to optimize the S Pen experience as well as improve the latency. Thanks to our point prediction technology powered by deep learning, we are able to predict the next movement of the S Pen to minimize the delay between input and output on the screen of the Z Fold3. This, in combination with the front buffer rendering technology and unique large screen, means that our users can enjoy the most natural handwriting, note-taking and drawing on their Z Fold3.

Q: How many languages does the S Pen now support? What were some of the difficulties of expanding the number of recognized languages for the Z Fold3’s S Pen?

At this moment, the S Pen supports 88 languages in total, 14 of which were newly added in the past year.

Adding a new language for handwriting recognition requires a lot of effort. In order to achieve the world’s best level of recognition quality and speed, we have developed an advanced AI technology. This technology is based on deep learning across millions of users’ samples and language modeling for natural language processing (NLP).

Q: Do you have any special tips for making the most out of the S Pen on the Z Fold3 to share with users?

For those who love writing by hand, the S Pen To Text feature now supports writing with the S Pen directly into text fields. The handwriting is then automatically converted to text. Also, you can edit any printed text with your S Pen – for example, when correcting spacing or inserting words.

One feature that will also interest many users is Neat Shape mode. It can help you to not only draw shapes, but also build a flowchart.

Make sure to check the Tips application and not skip usage tips – there you will discover the newest S Pen and input features, and also learn how to use many other Samsung apps to their fullest potential.

Q: Looking forward, what do you think the future of the S Pen looks like? How might it evolve?

We hope to introduce more ways to support better user experiences by providing even more personalization in the near future. Our mission is to make S Pen input like writing in your notepad with a pen, but better and smarter.

Most of all, I want for as many people as possible to experience the power and convenience of the S Pen and its handwritten input capabilities. It has the potential to enhance input on a wide range of devices. We view the S Pen not only as a cool gadget, but first and foremost as an accessible and useful tool. I hope I will see the S Pen and its features casually used in education, in the workplace and in everyday life. I do believe we are set to see a revolution in input technology.

The full interview video can be viewed here:

In this relay series, Samsung Newsroom is introducing tech experts from Samsung’s R&D centers around the globe to hear more about the work they do and the ways in which it is directly improving the lives of consumers.

The second expert in the series is Lukasz Slabinski, Head of the Artificial Intelligence Team at Samsung R&D Institute Poland (SRPOL). Slabinski joined SRPOL in 2013 as a Senior Engineer, and following 8 years of dedicated work, now leads the AI Team at SRPOL. Read on to hear more about the exciting innovation Slabinski and his team are involved with at SRPOL.

Q: Designing solutions for the speech recognition field is known to be highly intricate. When working on language-related technologies, what challenges have you encountered and how have you been overcoming them?

In my opinion, language-related technologies are far more complex than any other ones. Humankind communicates in almost 7000 constantly evolving languages, sub-divided into endless accents and dialects. Moreover, human language is far less objective than, for example, a picture, which can be described in mathematical formulas. People encode their thoughts as a set of sounds or characters into a message, which then needs to be decoded and interpreted by others. Because each phase of this process is personal, creative and non-deterministic, language-based human communication is very complex and ambiguous. Thus, on the one hand, we can enjoy beautiful poetry and funny jokes, and on the other, occasionally suffer from misunderstandings.

The R&D people who work on natural language processing (NLP) often reach their own, innately human, limitations. Even we encounter issues communicating clearly with colleagues at work, or family at home. So how, for example, can an engineer who speaks 2 languages design and code a machine translation system for 40 different languages? We solve this paradox using machine learning technologies.

During the process known as ‘training’, we automatically extract general patterns based on examples from our datasets and memorize them in the form of a model. To build a machine translation system, we train a neural network to map a sentence in different languages based on millions of examples, all carefully collected and cleaned beforehand. It sounds easy, but we deal here with 3 fundamental challenges.

The first challenge is the design of an appropriate machine learning model architecture capable of memorizing and generalizing enough language patterns for given problems such as machine translation, sentiment analysis, text summarization and others.

The second challenge is the preparation of sufficient amount of training data, as machine learning systems can recognize and memorize only those patterns presented in the training dataset.

The final challenge is the deployment of an already-trained machine learning model onto a dedicated Cloud or on-device platform.

We address these challenges by harnessing the vast expertise of our engineers, sophisticated approaches to collecting data and through endless experimentation with the state-of-the-art machine learning architectures.

Q: Can you please briefly introduce your AI Team, the Samsung R&D Institute Poland (SRPOL) and the kind of work that goes on there?

SRPOL is one of the largest international software R&D centers in Poland. It is located in two cities: Warsaw, the capital city of Poland and Cracow which is a major technology hub in its region. We closely collaborate with local start-ups, universities and research institutions.

The mission of the AI Team at SRPOL is the creation of the AI-based features, tools and services capable of facilitating and enriching human lives. We mainly focus on the NLP and Audio Intelligence areas, but we also possess expertise across many different specialties, including recommendation systems, indoor positioning, visual analytics and AR.

Q: As the head of the Polish Institute’s AI Team since 2018, you have overseen a myriad of projects both with and without the NLP focus. What are you and your team working on now?

Regarding the NLP area, we have been continuing our journey that began over 10 years ago by the development of systems such as Machine Translation, Dialogue Systems including Question Answering and Text Analytics. We work both on scalable, powerful cloud-based services as well as on fast and offline working on-device applications.

Audio Intelligence is a newer area for us. We began to focus our research capabilities on it around several years ago as the area had been gaining importance. Currently, we work on sound recognition, separation, enhancement and analysis. During our work, we take all levels of audio processing into consideration, from acoustic scene understanding to the fine-tuning of the embedded audio algorithms on devices with very limited hardware resources, such as wireless earbuds.

Q: Your technological focuses include NLP, text & data mining, audio intelligence and more. Has your research directly affected the development of any specific Samsung product or service, and what benefit has your team’s contribution offered to users?

SRPOL has a long record of commercializing AI technologies, but we did not do it alone. We are proud to be a part of a bigger picture, wherein SRPOL works closely with other Samsung R&D centers and contributes to commercialization.

For example, we contributed to the development of several intelligent text entry features for Samsung’s mobile devices, including the on-screen keyboard, hashtag feature, Samsung Note title recommendation and smart text replies on smartwatches.

We also contributed to the Galaxy Store’s Recommendation System, which suggests the most interesting games to a user based on their preferences.

Q: As an advocate for the new AI fields such as audio intelligence, what do you see as the main trends within your industry right now? How will this technology affect people’s daily lives?

I do believe that audio intelligence will be the next game-changer for all consumer electronic devices. Working on audio analytics is extremely important, as it is the missing part in advanced, truly human-centered AI-based systems.

Powerful NLP systems analyze the user’s intent as expressed by text and speech. Computer vision algorithms are behind almost every camera and visual content’s output. For most of us, it is hard to imagine driving a car without navigation, typing a message without spelling correctors, or searching for information without the Internet. But, except for a few professional applications, so far, we very rarely use intelligent audio technology to enhance our hearing. In my opinion, this is set to change soon.

Let’s imagine that we have a commonly available technology that allows people to select what and how they want to hear. For example, during a lunch with a friend in a park located in a busy city center, someone could choose to hear only the sounds of nature and the person they are speaking with. Or, let’s imagine an advanced VR or AR system, recently referred to as Metaverse that creates an immersive 3D audio experience directly in people’s heads. Just these two concepts generate hundreds of new possible use cases, but let’s go further. How about hearing things that are currently inaudible to people? Now humans can hear only a narrow spectrum of different sounds. Our world is full of meaningful sounds which, for the most part, the current AI technologies are not involved in. With the development of the audio intelligence technologies, I believe that all of this is going to affect people’s lives hugely.

▲ Researchers at Samsung R&D Institute Poland work on Active Noise Cancellation (ANC) technology development with a Head & Torso Simulator (HATS) in an anechoic room.

Q: How have you been incorporating the current trends into the research you do at Samsung R&D Institute Poland?

Aside from NLP and Audio, we are also working to find the most effective ways to build truly multimodal systems. To do that, we proceed with research and analyzing use cases from different perspectives. Such analysis is made possible thanks to our diverse and interdisciplinary team that consists of engineers, linguists, data scientists and more.

Q: What has been your most important achievement at SRPOL so far?

That would be our Machine Translation solution. Our solution has garnered wins at various competitions for five years straight: the International Workshop on Spoken Language Translation (IWSLT) from 2017 to 2020; the Workshop on Machine Translation (WMT) in 2020; and the Workshop on Asian Translation (WAT) in 2021. These are among the most prestigious international competitions in our field.

Winning recognition at WAT this year was a particularly satisfying milestone, as developing our solution for the Asian languages was originally a difficult feat for us as Polish engineers – but this achievement has proven the true power of our technology that goes beyond a mere demo showcase.

Another achievement that I am very proud of is the speed of growth that the audio intelligence team and its technology development have achieved. In just a few years, after starting pretty much from scratch, we were able to stand on the podium of the workshop on Detection and Classification of Acoustic Scenes and Events for two consecutive years, 2019 and 2020. We have also published several scientific papers and patents in this area. I am sure this is just the beginning of our prolific activities in this field.

An interview with Bin Dai, a machine learning expert from Samsung Research Institute China-Beijing can be found in the following episode.

This year’s AI Forum, the fourth of its kind, is being held over two days this November 2 and 3. The second day of the event, hosted by Samsung Research, the advanced R&D hub of the company that leads the development of future technologies for Samsung Electronics’ SET (end-products) business, facilitated discussion around how industry experts and academics alike can further research into AI technologies, products, and services that directly impact and enhance the lives of all people.

AI Forum Day 2: Human-Centered AI

To open up the second day of the AI Forum 2020 on November 3, Dr. Sebastian Seung, President and Head of Samsung Research, gave a welcome speech that highlighted how the theme of the Forum’s second day, “Human-Centered AI,” is very much in line with Samsung’s vision of creating products and services that make all our lives richer and more convenient.

Dr. Seung outlined the importance of collaboration between corporate and academic AI researchers. “Unlike academic researchers, who have greater freedom to explore their professional interests, corporate researchers dealing with real-world issues often encounter constraints in their research,” explained Dr. Seung. “To overcome such restraints, companies are driven to find creative ways to problem-solve and to conduct truly innovative research.”

Dr. Seung went on to outline the areas in which Samsung has been progressing its AI research, highlighting how the company has been expanding its research into traditional AI fields such as vision and graphics, speech and language and robotics. He noted that the company has also been making great efforts with their on-device AI, with work being done to develop how AI functions on devices with limited computational power, limited electrical power consumption and other such constraints. He also highlighted the company’s focus on the field of health and wellness, stressing it as a very fascinating area wherein AI, data and devices can come together to benefit people in their health and wellness journeys.

In order to showcase the big picture within which AI research exists, Dr. Seung then presented a range of Samsung products that are infused with AI technologies, noting the existing technical challenges that Samsung and other AI researchers around the world should be looking to surmount in order to extend the capability of AI as much as possible to help people. “AI research for a better world only begins when we think deeply about how AI is capable of improving our lives and changing human behavior for the better,” concluded Dr. Seung.

Expert Highlights: Keynote Speeches

For the second day of the Samsung AI Forum 2020, some of the most prolific experts in AI worldwide were invited to participate in the Forum’s lectures and discussions. Professor Christopher Manning of Stanford University, a world-renowned scholar in the field of natural language processing (NLP), gave a presentation titled Natural Language Understanding and Conversational AI. Professor Manning shared the current status and latest trends in NLP technologies, highlighting the recent rapid development of such technologies, and introduced more accurate conversational agents and more effective open domain social robots based on them. Professor Devi Parikh of the Georgia Institute of Technology gave a lecture titled Multimodal and Creative AI Systems, in which she described her work into computer vision systems that humans can interact with via language and AI systems that can assist humans with their creative and artistic endeavors.

Professor Subbarao Kambhampati of Arizona State University, a founding board member of the nonprofit The Partnership on AI, gave a lecture titled Synthesizing Interpretable Behavior for Human-Aware AI Systems. Using several case studies from his research, Professor Kambhampati highlighted the growing need for AI systems to work synergistically with humans in everyday life and asserted that, for this to happen, the systems need to exhibit behavior interpretable by humans.

Lastly, Daniel D. Lee, Executive Vice President of Samsung Research, Head of Samsung’s AI Center in New York and Professor at Cornell Tech, delivered a lecture under the theme AI for Robots and People. He examined the technologies being used in the latest machine learning algorithms and explained how they can be used both to develop more advanced robotics systems and to improve people’s everyday lives.

Stay tuned to Samsung Newsroom for more information on the Samsung AI Forum 2020.